[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:17:06 dist-test-slave-clmx krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:17:06 dist-test-slave-clmx krb5kdc[399](info): set up 2 sockets
Sep 25 12:17:06 dist-test-slave-clmx krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 12:17:07.709375 392 mini_kdc.cc:121] Time spent starting KDC: real 2.599s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 12:17:07.894936 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.185s user 0.001s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260925 12:17:08.022562 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.127s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260925 12:17:08.123934 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.101s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/test-user.keytab.
W20260925 12:17:08.921363 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.797s user 0.000s sys 0.002s
Sep 25 12:17:08 dist-test-slave-clmx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:17:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260925 12:17:09.275333 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.334s user 0.000s sys 0.002s
I20260925 12:17:09.275866 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43495
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:09.354584 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:09.354763 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:09.354784 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:09.355947 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:09.355985 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:09.355999 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:09.356010 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:09.356055 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:17:09.357318 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:09.357476 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:09.357699 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:09.362287 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:09.362288 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:09.362317 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:09.362797 412 server_base.cc:1034] running on GCE node
I20260925 12:17:09.363022 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:09.363300 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:09.364463 412 hybrid_clock.cc:648] HybridClock initialized: now 1790338629364436 us; error 40 us; skew 500 ppm
Sep 25 12:17:09 dist-test-slave-clmx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:09.366735 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:17:09.367179 412 webserver.cc:460] Webserver started at http://127.0.98.62:35261/ using document root <none> and password file <none>
I20260925 12:17:09.367399 412 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:09.367477 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:09.367599 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:09.368517 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "c6ac15118f2a4220ac6f8ddd701b1825"
format_stamp: "Formatted at 2026-09-25 12:17:09 on dist-test-slave-clmx"
server_key: "e2a088771a26217e3fe034408c905041"
server_key_iv: "b624f92044f9e92bb1753fade35f42e8"
server_key_version: "encryptionkey@0"
I20260925 12:17:09.368870 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "c6ac15118f2a4220ac6f8ddd701b1825"
format_stamp: "Formatted at 2026-09-25 12:17:09 on dist-test-slave-clmx"
server_key: "e2a088771a26217e3fe034408c905041"
server_key_iv: "b624f92044f9e92bb1753fade35f42e8"
server_key_version: "encryptionkey@0"
I20260925 12:17:09.370086 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:17:09.370786 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:09.370960 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:17:09.371068 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "c6ac15118f2a4220ac6f8ddd701b1825"
format_stamp: "Formatted at 2026-09-25 12:17:09 on dist-test-slave-clmx"
server_key: "e2a088771a26217e3fe034408c905041"
server_key_iv: "b624f92044f9e92bb1753fade35f42e8"
server_key_version: "encryptionkey@0"
I20260925 12:17:09.371153 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:09.397119 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:09.398957 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:09.399175 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:09.403116 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43495
I20260925 12:17:09.403155 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43495 every 8 connection(s)
I20260925 12:17:09.403483 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:09.404313 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:09.406515 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825: Bootstrap starting.
I20260925 12:17:09.407020 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:09.407277 480 log.cc:824] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:09.407969 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825: No bootstrap required, opened a new log
I20260925 12:17:09.409420 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6ac15118f2a4220ac6f8ddd701b1825" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43495 } }
I20260925 12:17:09.409547 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:09.409570 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ac15118f2a4220ac6f8ddd701b1825, State: Initialized, Role: FOLLOWER
I20260925 12:17:09.409741 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6ac15118f2a4220ac6f8ddd701b1825" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43495 } }
I20260925 12:17:09.409824 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:09.409847 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:09.409876 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:09.410334 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6ac15118f2a4220ac6f8ddd701b1825" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43495 } }
I20260925 12:17:09.410341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 412
I20260925 12:17:09.410440 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:17:09.410419 480 leader_election.cc:302] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6ac15118f2a4220ac6f8ddd701b1825; no voters:
I20260925 12:17:09.410681 480 leader_election.cc:288] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:17:09.410703 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:09.410861 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825 [term 1 LEADER]: Becoming Leader. State: Replica: c6ac15118f2a4220ac6f8ddd701b1825, State: Running, Role: LEADER
I20260925 12:17:09.410971 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:09.411011 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6ac15118f2a4220ac6f8ddd701b1825" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43495 } }
I20260925 12:17:09.411226 392 external_mini_cluster.cc:1428] Setting key c88aa25d300c0b5415ca1e6aa6ba7a6b
I20260925 12:17:09.411417 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6ac15118f2a4220ac6f8ddd701b1825. Latest consensus state: current_term: 1 leader_uuid: "c6ac15118f2a4220ac6f8ddd701b1825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6ac15118f2a4220ac6f8ddd701b1825" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43495 } } }
I20260925 12:17:09.411504 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:09.411556 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6ac15118f2a4220ac6f8ddd701b1825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6ac15118f2a4220ac6f8ddd701b1825" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43495 } } }
I20260925 12:17:09.411669 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:09.411983 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:09.412582 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:17:09.414229 491 catalog_manager.cc:1347] Generated new cluster ID: d993d16b36ad49ae9cf03977a2e9040b
I20260925 12:17:09.414286 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:17:09 dist-test-slave-clmx krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:09.426093 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:09.412439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.0.98.62:43495)
0925 12:17:09.412618 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:09.412627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:09.412779 (+ 152us) server_negotiation.cc:407] Connection header received
0925 12:17:09.414853 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:09.414858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:09.414912 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:09.415036 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:09.416954 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:09.417809 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:09.418830 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:09.419205 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:09.419313 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:09.422500 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:09.422508 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:09.422509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:09.422517 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:09.424167 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:09.424635 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:09.424638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:09.424639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:09.424735 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:09.424999 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:09.425000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:09.425000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:09.425224 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:09.425394 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:09.425673 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 12:17:09.425789 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260925 12:17:09.427069 491 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:17:09.427526 491 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:09.430182 491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6ac15118f2a4220ac6f8ddd701b1825: Generated new TSK 0
I20260925 12:17:09.430336 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:17:09.458995 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43495
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:09.561492 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:09.561676 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:09.561699 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:09.562896 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:09.562937 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:09.562963 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:17:09.564531 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:09.564770 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:09.565011 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:09.570714 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:09.570781 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:09.570904 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:09.570863 506 server_base.cc:1034] running on GCE node
I20260925 12:17:09.571257 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:09.571516 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:09.572693 506 hybrid_clock.cc:648] HybridClock initialized: now 1790338629572681 us; error 53 us; skew 500 ppm
Sep 25 12:17:09 dist-test-slave-clmx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:09.575455 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:17:09.575937 506 webserver.cc:460] Webserver started at http://127.0.98.1:41187/ using document root <none> and password file <none>
I20260925 12:17:09.576198 506 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:09.576274 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:09.576424 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:09.577409 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "4dc3c973415546a18071f1d6eea7b93b"
format_stamp: "Formatted at 2026-09-25 12:17:09 on dist-test-slave-clmx"
server_key: "ff815ebf148b26f0d77bba3227fb6c28"
server_key_iv: "7b47260df041bedd7e18006325539481"
server_key_version: "encryptionkey@0"
I20260925 12:17:09.577764 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4dc3c973415546a18071f1d6eea7b93b"
format_stamp: "Formatted at 2026-09-25 12:17:09 on dist-test-slave-clmx"
server_key: "ff815ebf148b26f0d77bba3227fb6c28"
server_key_iv: "7b47260df041bedd7e18006325539481"
server_key_version: "encryptionkey@0"
I20260925 12:17:09.579073 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:17:09.579802 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:09.579975 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:17:09.580067 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "4dc3c973415546a18071f1d6eea7b93b"
format_stamp: "Formatted at 2026-09-25 12:17:09 on dist-test-slave-clmx"
server_key: "ff815ebf148b26f0d77bba3227fb6c28"
server_key_iv: "7b47260df041bedd7e18006325539481"
server_key_version: "encryptionkey@0"
I20260925 12:17:09.580145 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:09.587901 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:09.589993 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:09.590142 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:09.590373 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:09.590647 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:09.590706 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:09.590744 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:09.590770 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:09.596726 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43431
I20260925 12:17:09.596798 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43431 every 8 connection(s)
I20260925 12:17:09.597103 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 12:17:09 dist-test-slave-clmx krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:09.604023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 506
I20260925 12:17:09.604135 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:17:09.604285 392 external_mini_cluster.cc:1428] Setting key d5ab74953ea10cdafd5190180dd14602
I20260925 12:17:09.605489 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:09.597814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34485 (local address 127.0.98.62:43495)
0925 12:17:09.597877 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:09.597879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:09.598064 (+ 185us) server_negotiation.cc:407] Connection header received
0925 12:17:09.598440 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:09.598441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:09.598449 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:09.598497 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:09.599278 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:09.599819 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:09.600748 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:09.600932 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:09.600981 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:09.602833 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:09.602834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:09.602834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:09.602840 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:09.604014 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:09.604515 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:09.604517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:09.604517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:09.604580 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:09.604847 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:09.604848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:09.604849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:09.604999 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:09.605066 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:09.605324 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 12:17:09.605399 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 12:17:09.605666 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:09.597776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43495 (local address 127.0.98.1:34485)
0925 12:17:09.597959 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:09.597994 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:09.598368 (+ 374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:09.598538 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:09.598540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:09.599007 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:09.599202 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:09.599206 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:09.599893 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:09.599894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:09.600675 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:09.600677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:09.601003 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:09.601003 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:09.601096 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:09.601465 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:09.601472 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:09.602723 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:09.604092 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:09.604099 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:09.604101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:09.604436 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:09.604664 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:09.604665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:09.604665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:09.604775 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:09.605057 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:09.605063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:09.605195 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 12:17:09.605303 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 12:17:09.605444 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
I20260925 12:17:09.606056 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43495
I20260925 12:17:09.606143 635 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:09.606295 635 heartbeater.cc:511] Master 127.0.98.62:43495 requested a full tablet report, sending...
I20260925 12:17:09.606758 444 ts_manager.cc:194] Registered new tserver with Master: 4dc3c973415546a18071f1d6eea7b93b (127.0.98.1:43431)
I20260925 12:17:09.607470 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34485
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:17:09.628250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43495
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:09.707336 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:09.707506 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:09.707527 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:09.708603 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:09.708644 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:09.708669 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:17:09.709968 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:09.710129 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:09.710276 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:09.715230 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:09.715233 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:09.715387 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:09.715490 640 server_base.cc:1034] running on GCE node
I20260925 12:17:09.715694 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:09.715914 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:09.717060 640 hybrid_clock.cc:648] HybridClock initialized: now 1790338629717051 us; error 25 us; skew 500 ppm
Sep 25 12:17:09 dist-test-slave-clmx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:09.719321 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:17:09.719795 640 webserver.cc:460] Webserver started at http://127.0.98.2:45067/ using document root <none> and password file <none>
I20260925 12:17:09.720005 640 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:09.720043 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:09.720181 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:09.721158 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "ca0fea3ef63c4fb8949c44af973f5fcd"
format_stamp: "Formatted at 2026-09-25 12:17:09 on dist-test-slave-clmx"
server_key: "0f3393374eb5c3f42e11f36c90bec36c"
server_key_iv: "0c5608e00eced44951f0399961d23dc8"
server_key_version: "encryptionkey@0"
I20260925 12:17:09.721498 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ca0fea3ef63c4fb8949c44af973f5fcd"
format_stamp: "Formatted at 2026-09-25 12:17:09 on dist-test-slave-clmx"
server_key: "0f3393374eb5c3f42e11f36c90bec36c"
server_key_iv: "0c5608e00eced44951f0399961d23dc8"
server_key_version: "encryptionkey@0"
I20260925 12:17:09.723245 640 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 12:17:09.723995 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:09.724182 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:09.724247 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "ca0fea3ef63c4fb8949c44af973f5fcd"
format_stamp: "Formatted at 2026-09-25 12:17:09 on dist-test-slave-clmx"
server_key: "0f3393374eb5c3f42e11f36c90bec36c"
server_key_iv: "0c5608e00eced44951f0399961d23dc8"
server_key_version: "encryptionkey@0"
I20260925 12:17:09.724359 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:09.738647 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:09.740314 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:09.740455 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:09.740671 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:09.740954 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:09.741012 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:09.741051 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:09.741076 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:09.746819 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42571
I20260925 12:17:09.746862 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42571 every 8 connection(s)
I20260925 12:17:09.747131 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:17:09 dist-test-slave-clmx krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:09.752796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 640
I20260925 12:17:09.752885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:17:09.753075 392 external_mini_cluster.cc:1428] Setting key 2519b91d649fe9de043bd946ba94e946
I20260925 12:17:09.755318 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:09.747798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54545 (local address 127.0.98.62:43495)
0925 12:17:09.747876 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:09.747877 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:09.748031 (+ 154us) server_negotiation.cc:407] Connection header received
0925 12:17:09.748401 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:09.748401 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:09.748409 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:09.748458 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:09.749199 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:09.749640 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:09.750445 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:09.750623 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:09.750671 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:09.752487 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:09.752489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:09.752489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:09.752494 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:09.753981 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:09.754426 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:09.754427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:09.754427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:09.754470 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:09.754701 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:09.754702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:09.754702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:09.754784 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:09.754828 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:09.755125 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 12:17:09.755211 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 12:17:09.755436 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:09.747781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43495 (local address 127.0.98.2:54545)
0925 12:17:09.747921 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:09.747955 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:09.748327 (+ 372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:09.748500 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:09.748503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:09.748979 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:09.749128 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:09.749132 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:09.749712 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:09.749713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:09.750378 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:09.750379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:09.750697 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:09.750697 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:09.750783 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:09.751157 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:09.751165 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:09.752407 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:09.754048 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:09.754053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:09.754055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:09.754357 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:09.754528 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:09.754528 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:09.754529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:09.754646 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:09.754841 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:09.754846 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:09.754981 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 12:17:09.755089 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 12:17:09.755232 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":84,"threads_started":1}
I20260925 12:17:09.755911 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43495
I20260925 12:17:09.756012 769 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:09.756155 769 heartbeater.cc:511] Master 127.0.98.62:43495 requested a full tablet report, sending...
I20260925 12:17:09.756474 444 ts_manager.cc:194] Registered new tserver with Master: ca0fea3ef63c4fb8949c44af973f5fcd (127.0.98.2:42571)
I20260925 12:17:09.756843 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54545
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:17:09.776607 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43495
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:09.851517 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:09.851752 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:09.851794 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:09.852880 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:09.852946 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:09.852990 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:17:09.854339 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:09.854573 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:09.854797 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:09.859252 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:09.859230 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:09.859243 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:09.859438 774 server_base.cc:1034] running on GCE node
I20260925 12:17:09.860085 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:09.860351 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:09.861517 774 hybrid_clock.cc:648] HybridClock initialized: now 1790338629861503 us; error 26 us; skew 500 ppm
Sep 25 12:17:09 dist-test-slave-clmx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:09.863756 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:17:09.864173 774 webserver.cc:460] Webserver started at http://127.0.98.3:38799/ using document root <none> and password file <none>
I20260925 12:17:09.864441 774 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:09.864512 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:09.864642 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:09.865603 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "6712d559069a4427a997a52d3c5a82dd"
format_stamp: "Formatted at 2026-09-25 12:17:09 on dist-test-slave-clmx"
server_key: "bedc4104a63cb4936baf63b5dcc5c25f"
server_key_iv: "1409c82e484fea8d2fbc93e6e5f80c8c"
server_key_version: "encryptionkey@0"
I20260925 12:17:09.865983 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6712d559069a4427a997a52d3c5a82dd"
format_stamp: "Formatted at 2026-09-25 12:17:09 on dist-test-slave-clmx"
server_key: "bedc4104a63cb4936baf63b5dcc5c25f"
server_key_iv: "1409c82e484fea8d2fbc93e6e5f80c8c"
server_key_version: "encryptionkey@0"
I20260925 12:17:09.867254 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:17:09.868019 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:09.868187 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:09.868280 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "6712d559069a4427a997a52d3c5a82dd"
format_stamp: "Formatted at 2026-09-25 12:17:09 on dist-test-slave-clmx"
server_key: "bedc4104a63cb4936baf63b5dcc5c25f"
server_key_iv: "1409c82e484fea8d2fbc93e6e5f80c8c"
server_key_version: "encryptionkey@0"
I20260925 12:17:09.868364 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:09.879244 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:09.881012 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:09.881170 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:09.881376 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:09.881659 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:09.881719 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:09.881767 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:09.881791 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:09.888482 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38253
I20260925 12:17:09.888513 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38253 every 8 connection(s)
I20260925 12:17:09.888811 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:17:09.890722 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 774
I20260925 12:17:09.890820 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:17:09.890986 392 external_mini_cluster.cc:1428] Setting key 94f66b2e8c169eb94185499ff6efe875
Sep 25 12:17:09 dist-test-slave-clmx krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:09.897540 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:09.889479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54295 (local address 127.0.98.62:43495)
0925 12:17:09.889542 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:09.889544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:09.889716 (+ 172us) server_negotiation.cc:407] Connection header received
0925 12:17:09.890124 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:09.890125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:09.890134 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:09.890186 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:09.891310 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:09.892090 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:09.893013 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:09.893181 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:09.893234 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:09.895051 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:09.895053 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:09.895053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:09.895059 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:09.896496 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:09.896828 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:09.896829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:09.896829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:09.896874 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:09.897074 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:09.897074 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:09.897074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:09.897158 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:09.897202 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:09.897419 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 12:17:09.897484 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 12:17:09.897711 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:09.889457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43495 (local address 127.0.98.3:54295)
0925 12:17:09.889601 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:09.889637 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:09.890046 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:09.890235 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:09.890238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:09.890944 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:09.891211 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:09.891218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:09.892177 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:09.892184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:09.892939 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:09.892941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:09.893249 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:09.893250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:09.893336 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:09.893722 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:09.893729 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:09.894968 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:09.896561 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:09.896565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:09.896567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:09.896764 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:09.896942 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:09.896943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:09.896943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:09.897024 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:09.897210 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:09.897214 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:09.897304 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 12:17:09.897375 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 12:17:09.897504 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":84,"threads_started":1}
I20260925 12:17:09.898157 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43495
I20260925 12:17:09.898247 903 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:09.898416 903 heartbeater.cc:511] Master 127.0.98.62:43495 requested a full tablet report, sending...
I20260925 12:17:09.898772 444 ts_manager.cc:194] Registered new tserver with Master: 6712d559069a4427a997a52d3c5a82dd (127.0.98.3:38253)
I20260925 12:17:09.899068 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54295
I20260925 12:17:09.902908 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 12:17:09 dist-test-slave-clmx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338629, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 12:17:09 dist-test-slave-clmx krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338629, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:17:09.924773 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:09.917241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39912 (local address 127.0.98.1:43431)
0925 12:17:09.917410 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:09.917415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:09.917426 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:17:09.917461 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:09.917462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:09.917488 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:09.917552 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:09.918027 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:09.918767 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:09.919460 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:09.919815 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:09.919870 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:09.922063 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:09.922071 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:09.922071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:09.922078 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:09.923446 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:09.923879 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:09.923883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:09.923883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:09.923942 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:09.924160 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:09.924161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:09.924161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:09.924305 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:09.924489 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:09.924575 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 12:17:09.924650 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
W20260925 12:17:09.925132 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39912
Sep 25 12:17:09 dist-test-slave-clmx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 12:17:09 dist-test-slave-clmx krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:17:09.946720 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:09.939989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.0.98.1:43431)
0925 12:17:09.940022 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:09.940024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:09.940174 (+ 150us) server_negotiation.cc:407] Connection header received
0925 12:17:09.940244 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:09.940245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:09.940256 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:09.940308 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:09.940860 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:09.941329 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:09.942072 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:09.942280 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:09.942373 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:09.944460 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:09.944462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:09.944463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:09.944469 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:09.945608 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:09.945943 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:09.945944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:09.945944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:09.945996 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:09.946206 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:09.946207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:09.946208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:09.946311 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:09.946460 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:09.946560 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 12:17:09.946631 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 12:17:09.947182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 506
I20260925 12:17:09.951341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 640
I20260925 12:17:09.956568 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 774
I20260925 12:17:09.960647 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 412
2026-09-25T12:17:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4875 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:17:10 dist-test-slave-clmx krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:17:10 dist-test-slave-clmx krb5kdc[917](info): set up 2 sockets
Sep 25 12:17:10 dist-test-slave-clmx krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260925 12:17:10.947057 392 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:17:11 dist-test-slave-clmx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:17:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:17:11.056304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35803
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:11.134392 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:11.134641 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:11.134677 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:11.135883 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:11.135950 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:11.135973 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:11.135995 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:11.136022 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:17:11.137444 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:11.137722 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:11.137935 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:11.142478 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:11.142488 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:11.142493 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:11.142772 930 server_base.cc:1034] running on GCE node
I20260925 12:17:11.142977 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:11.143232 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:11.144387 930 hybrid_clock.cc:648] HybridClock initialized: now 1790338631144363 us; error 43 us; skew 500 ppm
Sep 25 12:17:11 dist-test-slave-clmx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:11.146675 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:17:11.147090 930 webserver.cc:460] Webserver started at http://127.0.98.62:36175/ using document root <none> and password file <none>
I20260925 12:17:11.147262 930 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:11.147300 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:11.147380 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:11.148399 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "df714160b60c44eb9144412ca8dd0acf"
format_stamp: "Formatted at 2026-09-25 12:17:11 on dist-test-slave-clmx"
server_key: "6ee918715b10c235acbcedb7ee06009e"
server_key_iv: "cf8e09efb0ebf60d2519060cc10fc7cc"
server_key_version: "encryptionkey@0"
I20260925 12:17:11.148685 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "df714160b60c44eb9144412ca8dd0acf"
format_stamp: "Formatted at 2026-09-25 12:17:11 on dist-test-slave-clmx"
server_key: "6ee918715b10c235acbcedb7ee06009e"
server_key_iv: "cf8e09efb0ebf60d2519060cc10fc7cc"
server_key_version: "encryptionkey@0"
I20260925 12:17:11.149845 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:17:11.150478 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:11.150642 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:17:11.150748 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "df714160b60c44eb9144412ca8dd0acf"
format_stamp: "Formatted at 2026-09-25 12:17:11 on dist-test-slave-clmx"
server_key: "6ee918715b10c235acbcedb7ee06009e"
server_key_iv: "cf8e09efb0ebf60d2519060cc10fc7cc"
server_key_version: "encryptionkey@0"
I20260925 12:17:11.150799 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:11.178540 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:11.180531 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:11.180620 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:11.184592 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35803
I20260925 12:17:11.184654 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35803 every 8 connection(s)
I20260925 12:17:11.184962 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:11.185787 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:11.187916 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf: Bootstrap starting.
I20260925 12:17:11.188467 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:11.188777 998 log.cc:824] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:11.189438 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf: No bootstrap required, opened a new log
I20260925 12:17:11.190555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 930
I20260925 12:17:11.190639 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:17:11.190815 392 external_mini_cluster.cc:1428] Setting key 44c3325b713ae81f8696c79dc42c2ab4
I20260925 12:17:11.191514 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df714160b60c44eb9144412ca8dd0acf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35803 } }
I20260925 12:17:11.191706 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:11.191777 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df714160b60c44eb9144412ca8dd0acf, State: Initialized, Role: FOLLOWER
I20260925 12:17:11.191907 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df714160b60c44eb9144412ca8dd0acf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35803 } }
I20260925 12:17:11.191987 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:11.192008 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:11.192058 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:11.192539 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df714160b60c44eb9144412ca8dd0acf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35803 } }
I20260925 12:17:11.192616 998 leader_election.cc:302] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df714160b60c44eb9144412ca8dd0acf; no voters:
I20260925 12:17:11.192792 998 leader_election.cc:288] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:17:11.192844 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:11.192989 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf [term 1 LEADER]: Becoming Leader. State: Replica: df714160b60c44eb9144412ca8dd0acf, State: Running, Role: LEADER
I20260925 12:17:11.193017 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:11.194255 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df714160b60c44eb9144412ca8dd0acf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35803 } }
I20260925 12:17:11.194801 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf [sys.catalog]: SysCatalogTable state changed. Reason: New leader df714160b60c44eb9144412ca8dd0acf. Latest consensus state: current_term: 1 leader_uuid: "df714160b60c44eb9144412ca8dd0acf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df714160b60c44eb9144412ca8dd0acf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35803 } } }
I20260925 12:17:11.194924 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df714160b60c44eb9144412ca8dd0acf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df714160b60c44eb9144412ca8dd0acf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35803 } } }
I20260925 12:17:11.195264 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:11.195266 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf [sys.catalog]: This master's current role is: LEADER
W20260925 12:17:11.195530 1017 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:17:11.195578 1017 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:17:11.195894 1020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:11.196062 1020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:17:11.197368 1020 catalog_manager.cc:1347] Generated new cluster ID: 03e6f44ed190460fa317963d6841f93c
I20260925 12:17:11.197417 1020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:17:11 dist-test-slave-clmx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:11.201715 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.191344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44732 (local address 127.0.98.62:35803)
0925 12:17:11.191506 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:11.191514 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:11.191548 (+ 34us) server_negotiation.cc:407] Connection header received
0925 12:17:11.191682 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:11.191689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:11.191742 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:11.191882 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:11.192803 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.194065 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.195117 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.195715 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.195800 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.198355 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:11.198358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:11.198358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:11.198368 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:11.200097 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:11.200463 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:11.200465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:11.200465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:11.200575 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:11.200806 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:11.200807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:11.200807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:11.201067 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:11.201195 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:11.201398 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 12:17:11.201520 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260925 12:17:11.221805 1020 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:17:11.222482 1020 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:11.229602 1020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df714160b60c44eb9144412ca8dd0acf: Generated new TSK 0
I20260925 12:17:11.229746 1020 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:17:11.258349 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35803
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:11.337960 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:11.338150 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:11.338219 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:11.339418 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:11.339460 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:11.339488 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:17:11.340925 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35803
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:11.341166 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:11.341396 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:11.347474 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:11.347498 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:11.347707 1024 server_base.cc:1034] running on GCE node
W20260925 12:17:11.349010 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:11.349255 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:11.349515 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:11.350654 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790338631350616 us; error 36 us; skew 500 ppm
Sep 25 12:17:11 dist-test-slave-clmx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:11.353663 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:17:11.354142 1024 webserver.cc:460] Webserver started at http://127.0.98.1:36489/ using document root <none> and password file <none>
I20260925 12:17:11.354424 1024 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:11.354490 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:11.354609 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:11.356148 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "47c0e0b4f03f4cc7b84589e3c341fe12"
format_stamp: "Formatted at 2026-09-25 12:17:11 on dist-test-slave-clmx"
server_key: "47e77323f6e3cc7e0cdc30fa1350df04"
server_key_iv: "9380cea9615d809a310dab3e316f3ec2"
server_key_version: "encryptionkey@0"
I20260925 12:17:11.356562 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "47c0e0b4f03f4cc7b84589e3c341fe12"
format_stamp: "Formatted at 2026-09-25 12:17:11 on dist-test-slave-clmx"
server_key: "47e77323f6e3cc7e0cdc30fa1350df04"
server_key_iv: "9380cea9615d809a310dab3e316f3ec2"
server_key_version: "encryptionkey@0"
I20260925 12:17:11.358323 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 12:17:11.359272 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:11.359441 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:11.359550 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "47c0e0b4f03f4cc7b84589e3c341fe12"
format_stamp: "Formatted at 2026-09-25 12:17:11 on dist-test-slave-clmx"
server_key: "47e77323f6e3cc7e0cdc30fa1350df04"
server_key_iv: "9380cea9615d809a310dab3e316f3ec2"
server_key_version: "encryptionkey@0"
I20260925 12:17:11.359645 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:11.387156 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:11.389762 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:11.389930 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:11.390206 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:11.390583 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:11.390651 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:11.390693 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:11.390733 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:11.397295 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34573
I20260925 12:17:11.397364 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34573 every 8 connection(s)
I20260925 12:17:11.397707 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:17:11.402763 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 1024
I20260925 12:17:11.402851 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:17:11.403015 392 external_mini_cluster.cc:1428] Setting key 6dcd5909dcc9e65426f61ad0397af52e
Sep 25 12:17:11 dist-test-slave-clmx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:11.407501 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.398419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37389 (local address 127.0.98.62:35803)
0925 12:17:11.398490 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:11.398491 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:11.398693 (+ 202us) server_negotiation.cc:407] Connection header received
0925 12:17:11.399141 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:11.399142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:11.399156 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:11.399217 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:11.400160 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.400641 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.401624 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.401840 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.401904 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.404252 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:11.404253 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:11.404254 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:11.404260 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:11.405959 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:11.406449 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:11.406449 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:11.406450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:11.406497 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:11.406824 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:11.406825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:11.406825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:11.406928 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:11.406975 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:11.407380 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 12:17:11.407441 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 12:17:11.407747 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.398378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35803 (local address 127.0.98.1:37389)
0925 12:17:11.398568 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:11.398607 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:11.399057 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:11.399287 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:11.399290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:11.399887 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:11.400076 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:11.400080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.400746 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.400746 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:11.401515 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:11.401517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.401921 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.401922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:11.402012 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.402452 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:11.402460 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:11.404163 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:11.406075 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:11.406081 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:11.406083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:11.406376 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:11.406584 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:11.406585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:11.406585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:11.406760 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:11.406993 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:11.406999 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:11.407170 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 12:17:11.407287 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 12:17:11.407473 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":111,"threads_started":1}
I20260925 12:17:11.408196 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35803
I20260925 12:17:11.408280 1153 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:11.408463 1153 heartbeater.cc:511] Master 127.0.98.62:35803 requested a full tablet report, sending...
I20260925 12:17:11.409034 962 ts_manager.cc:194] Registered new tserver with Master: 47c0e0b4f03f4cc7b84589e3c341fe12 (127.0.98.1:34573)
I20260925 12:17:11.409650 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37389
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:17:11.430637 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35803
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:11.510098 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:11.510325 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:11.510361 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:11.511533 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:11.511607 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:11.511694 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:17:11.513044 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35803
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:11.513286 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:11.513509 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:11.518167 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:11.518262 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:11.518198 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:11.518338 1158 server_base.cc:1034] running on GCE node
I20260925 12:17:11.518519 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:11.518764 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:11.519923 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790338631519907 us; error 42 us; skew 500 ppm
Sep 25 12:17:11 dist-test-slave-clmx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:11.522245 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:17:11.522732 1158 webserver.cc:460] Webserver started at http://127.0.98.2:41327/ using document root <none> and password file <none>
I20260925 12:17:11.522964 1158 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:11.523031 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:11.523159 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:11.524259 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "2b627538c484436288732d9333e425fd"
format_stamp: "Formatted at 2026-09-25 12:17:11 on dist-test-slave-clmx"
server_key: "e2c0bec11e409b6d55c787196287cf97"
server_key_iv: "8b99afc7ebaff6dc4e307a56bbac82dc"
server_key_version: "encryptionkey@0"
I20260925 12:17:11.524638 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2b627538c484436288732d9333e425fd"
format_stamp: "Formatted at 2026-09-25 12:17:11 on dist-test-slave-clmx"
server_key: "e2c0bec11e409b6d55c787196287cf97"
server_key_iv: "8b99afc7ebaff6dc4e307a56bbac82dc"
server_key_version: "encryptionkey@0"
I20260925 12:17:11.525887 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:17:11.526556 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:11.526744 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:11.526842 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "2b627538c484436288732d9333e425fd"
format_stamp: "Formatted at 2026-09-25 12:17:11 on dist-test-slave-clmx"
server_key: "e2c0bec11e409b6d55c787196287cf97"
server_key_iv: "8b99afc7ebaff6dc4e307a56bbac82dc"
server_key_version: "encryptionkey@0"
I20260925 12:17:11.526921 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:11.536891 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:11.538550 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:11.538709 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:11.538930 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:11.539234 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:11.539294 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:11.539335 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:11.539358 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:11.545259 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35071
I20260925 12:17:11.545306 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35071 every 8 connection(s)
I20260925 12:17:11.545601 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:17:11 dist-test-slave-clmx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:11.554457 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.546285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38353 (local address 127.0.98.62:35803)
0925 12:17:11.546371 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:11.546373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:11.546528 (+ 155us) server_negotiation.cc:407] Connection header received
0925 12:17:11.547012 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:11.547013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:11.547025 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:11.547076 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:11.548082 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.548568 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.549555 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.549774 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.549907 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.551906 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:11.551907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:11.551907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:11.551926 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:11.553038 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:11.553488 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:11.553489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:11.553490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:11.553549 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:11.553826 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:11.553827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:11.553827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:11.553944 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:11.554025 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:11.554286 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 12:17:11.554365 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 12:17:11.554632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 1158
I20260925 12:17:11.554672 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.546181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35803 (local address 127.0.98.2:38353)
0925 12:17:11.546395 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:11.546435 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:11.546904 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:11.547139 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:11.547155 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:11.547790 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:11.547980 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:11.547985 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.548710 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.548711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:11.549448 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:11.549450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.549912 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.549913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:11.550032 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.550447 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:11.550454 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:11.551787 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:11.553167 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:11.553172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:11.553174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:11.553396 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:11.553619 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:11.553620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:11.553620 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:11.553726 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:11.554017 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:11.554023 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:11.554157 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 12:17:11.554265 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 12:17:11.554426 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":119,"threads_started":1}
I20260925 12:17:11.554742 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:17:11.554906 392 external_mini_cluster.cc:1428] Setting key c8ea94eb346ab1477fedad3348ade5bd
I20260925 12:17:11.555313 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35803
I20260925 12:17:11.555431 1287 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:11.555641 1287 heartbeater.cc:511] Master 127.0.98.62:35803 requested a full tablet report, sending...
I20260925 12:17:11.556072 962 ts_manager.cc:194] Registered new tserver with Master: 2b627538c484436288732d9333e425fd (127.0.98.2:35071)
I20260925 12:17:11.556433 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38353
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:17:11.578380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35803
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:11.654048 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:11.654222 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:11.654243 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:11.655354 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:11.655393 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:11.655449 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:17:11.656934 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35803
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:11.657155 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:11.657361 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:11.661978 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:11.661979 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:11.662057 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:11.662235 1292 server_base.cc:1034] running on GCE node
I20260925 12:17:11.662386 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:11.662604 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:11.663753 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790338631663737 us; error 26 us; skew 500 ppm
Sep 25 12:17:11 dist-test-slave-clmx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:11.665904 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:17:11.666338 1292 webserver.cc:460] Webserver started at http://127.0.98.3:37349/ using document root <none> and password file <none>
I20260925 12:17:11.666543 1292 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:11.666617 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:11.666764 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:11.667867 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "7d4df22d924f4a5abe2959b079b21eb6"
format_stamp: "Formatted at 2026-09-25 12:17:11 on dist-test-slave-clmx"
server_key: "517939aa480807bc7d459faf23c454bf"
server_key_iv: "8b1ad8e752e10072ffa22a2acd14133d"
server_key_version: "encryptionkey@0"
I20260925 12:17:11.668223 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7d4df22d924f4a5abe2959b079b21eb6"
format_stamp: "Formatted at 2026-09-25 12:17:11 on dist-test-slave-clmx"
server_key: "517939aa480807bc7d459faf23c454bf"
server_key_iv: "8b1ad8e752e10072ffa22a2acd14133d"
server_key_version: "encryptionkey@0"
I20260925 12:17:11.669471 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:17:11.670107 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:11.670266 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:11.670356 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "7d4df22d924f4a5abe2959b079b21eb6"
format_stamp: "Formatted at 2026-09-25 12:17:11 on dist-test-slave-clmx"
server_key: "517939aa480807bc7d459faf23c454bf"
server_key_iv: "8b1ad8e752e10072ffa22a2acd14133d"
server_key_version: "encryptionkey@0"
I20260925 12:17:11.670433 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:11.686838 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:11.688566 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:11.688715 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:11.688918 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:11.689194 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:11.689268 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:11.689304 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:11.689325 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:11.695302 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43555
I20260925 12:17:11.695335 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43555 every 8 connection(s)
I20260925 12:17:11.695752 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 12:17:11 dist-test-slave-clmx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:11.702327 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 1292
I20260925 12:17:11.702422 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:17:11.702569 392 external_mini_cluster.cc:1428] Setting key 7b53138062222d96576fb58509ee7e95
I20260925 12:17:11.704319 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.696424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49989 (local address 127.0.98.62:35803)
0925 12:17:11.696466 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:11.696468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:11.696675 (+ 207us) server_negotiation.cc:407] Connection header received
0925 12:17:11.697075 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:11.697076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:11.697084 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:11.697147 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:11.697949 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.698415 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.699309 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.699477 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.699524 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.701471 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:11.701472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:11.701472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:11.701478 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:11.703071 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:11.703493 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:11.703495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:11.703495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:11.703543 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:11.703821 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:11.703822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:11.703822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:11.703917 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:11.703967 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:11.704191 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 12:17:11.704245 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:17:11.704515 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.696380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35803 (local address 127.0.98.3:49989)
0925 12:17:11.696529 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:11.696590 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:11.696998 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:11.697195 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:11.697199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:11.697700 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:11.697877 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:11.697880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.698495 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.698496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:11.699233 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:11.699234 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.699547 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.699548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:11.699711 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.700082 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:11.700089 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:11.701381 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:11.703147 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:11.703153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:11.703155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:11.703441 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:11.703604 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:11.703605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:11.703605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:11.703761 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:11.703975 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:11.703979 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:11.704085 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 12:17:11.704161 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 12:17:11.704264 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":91,"threads_started":1}
I20260925 12:17:11.705020 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35803
I20260925 12:17:11.705093 1421 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:11.705273 1421 heartbeater.cc:511] Master 127.0.98.62:35803 requested a full tablet report, sending...
I20260925 12:17:11.705677 962 ts_manager.cc:194] Registered new tserver with Master: 7d4df22d924f4a5abe2959b079b21eb6 (127.0.98.3:43555)
I20260925 12:17:11.705981 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49989
I20260925 12:17:11.714268 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:17:11.721403 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.715753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44744 (local address 127.0.98.62:35803)
0925 12:17:11.715792 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:11.715793 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:11.715888 (+ 95us) server_negotiation.cc:407] Connection header received
0925 12:17:11.715905 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:11.715906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:11.715913 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:11.715956 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:11.716451 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.716886 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.717579 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.717740 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.717786 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.719274 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:11.719275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:11.719275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:11.719282 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:11.720380 (+ 1098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:11.720654 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:11.720655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:11.720655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:11.720700 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:11.720887 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:11.720888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:11.720888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:11.721002 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:11.721080 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:11.721248 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 12:17:11.721301 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:17:11.723604 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:17:11.725191 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:17:11.731305 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.727318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34573 (local address 127.0.0.1:52574)
0925 12:17:11.727460 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:11.727493 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:11.727555 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:11.728329 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:11.728329 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:11.728331 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:11.728364 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:11.728365 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.729597 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.729598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:11.730327 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:11.730329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.731032 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.731033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:11.731108 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.731120 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 12:17:11.731189 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 12:17:11.731239 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":58,"threads_started":1}
I20260925 12:17:11.731586 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.727332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35071 (local address 127.0.0.1:49964)
0925 12:17:11.727663 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:11.727673 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:11.727725 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:11.727999 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:11.728000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:11.728001 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:11.728081 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:11.728083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.729397 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.729398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:11.730337 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:11.730339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.731334 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.731335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:11.731400 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.731404 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:17:11.731462 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 12:17:11.731498 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":3,"thread_start_us":41,"threads_started":1}
I20260925 12:17:11.731652 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.727584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49964 (local address 127.0.98.2:35071)
0925 12:17:11.727799 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:11.727802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:11.727811 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:17:11.727874 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:11.727876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:11.727891 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:11.727977 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:11.728215 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.729310 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.730443 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.731179 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.731247 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.731295 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:11.731498 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 12:17:11.731548 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":107,"threads_started":1}
I20260925 12:17:11.731674 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.727551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.0.98.3:43555)
0925 12:17:11.727736 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:11.727741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:11.727750 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:17:11.727785 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:11.727787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:11.727800 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:11.727871 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:11.728053 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.728948 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.730514 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.731179 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.731247 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.731295 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:11.731537 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 12:17:11.731573 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":90,"threads_started":1}
I20260925 12:17:11.731828 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.727332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43555 (local address 127.0.0.1:60954)
0925 12:17:11.727677 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:11.727685 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:11.727723 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:11.727917 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:11.727920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:11.727923 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:11.727980 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:11.727982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.729062 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.729063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:11.730444 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:11.730446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.731332 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.731334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:11.731404 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.731407 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:11.731453 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 12:17:11.731496 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":3,"thread_start_us":45,"threads_started":1}
I20260925 12:17:11.731338 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.727478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.0.98.1:34573)
0925 12:17:11.728105 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:11.728109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:11.728119 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:17:11.728161 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:11.728163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:11.728177 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:11.728249 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:11.728439 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.729498 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.730455 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.730921 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.731027 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.731085 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:11.731224 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 12:17:11.731263 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":77,"threads_started":1}
I20260925 12:17:11.732300 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb495007c89451fb06cb6aa32df86f1 (DEFAULT_TABLE table=test-table [id=60e2e1f23dee48f1b6df9d50c163aeac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:11.732466 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb495007c89451fb06cb6aa32df86f1 (DEFAULT_TABLE table=test-table [id=60e2e1f23dee48f1b6df9d50c163aeac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:11.732632 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb495007c89451fb06cb6aa32df86f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:11.732738 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb495007c89451fb06cb6aa32df86f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:11.736115 1436 tablet_bootstrap.cc:492] T 5fb495007c89451fb06cb6aa32df86f1 P 7d4df22d924f4a5abe2959b079b21eb6: Bootstrap starting.
I20260925 12:17:11.736577 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb495007c89451fb06cb6aa32df86f1 (DEFAULT_TABLE table=test-table [id=60e2e1f23dee48f1b6df9d50c163aeac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:11.736722 1436 tablet_bootstrap.cc:654] T 5fb495007c89451fb06cb6aa32df86f1 P 7d4df22d924f4a5abe2959b079b21eb6: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:11.736855 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb495007c89451fb06cb6aa32df86f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:11.737254 1436 log.cc:824] T 5fb495007c89451fb06cb6aa32df86f1 P 7d4df22d924f4a5abe2959b079b21eb6: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:11.737540 1435 tablet_bootstrap.cc:492] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd: Bootstrap starting.
I20260925 12:17:11.737978 1436 tablet_bootstrap.cc:492] T 5fb495007c89451fb06cb6aa32df86f1 P 7d4df22d924f4a5abe2959b079b21eb6: No bootstrap required, opened a new log
I20260925 12:17:11.738049 1436 ts_tablet_manager.cc:1397] T 5fb495007c89451fb06cb6aa32df86f1 P 7d4df22d924f4a5abe2959b079b21eb6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:17:11.738055 1435 tablet_bootstrap.cc:654] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:11.738472 1435 log.cc:824] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:11.739288 1435 tablet_bootstrap.cc:492] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd: No bootstrap required, opened a new log
I20260925 12:17:11.739375 1435 ts_tablet_manager.cc:1397] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:17:11.739988 1439 tablet_bootstrap.cc:492] T 5fb495007c89451fb06cb6aa32df86f1 P 47c0e0b4f03f4cc7b84589e3c341fe12: Bootstrap starting.
I20260925 12:17:11.739928 1436 raft_consensus.cc:348] T 5fb495007c89451fb06cb6aa32df86f1 P 7d4df22d924f4a5abe2959b079b21eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b627538c484436288732d9333e425fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35071 } } peers { permanent_uuid: "47c0e0b4f03f4cc7b84589e3c341fe12" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34573 } } peers { permanent_uuid: "7d4df22d924f4a5abe2959b079b21eb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43555 } }
I20260925 12:17:11.740098 1436 raft_consensus.cc:374] T 5fb495007c89451fb06cb6aa32df86f1 P 7d4df22d924f4a5abe2959b079b21eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:11.740144 1436 raft_consensus.cc:729] T 5fb495007c89451fb06cb6aa32df86f1 P 7d4df22d924f4a5abe2959b079b21eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4df22d924f4a5abe2959b079b21eb6, State: Initialized, Role: FOLLOWER
I20260925 12:17:11.740248 1436 consensus_queue.cc:260] T 5fb495007c89451fb06cb6aa32df86f1 P 7d4df22d924f4a5abe2959b079b21eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b627538c484436288732d9333e425fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35071 } } peers { permanent_uuid: "47c0e0b4f03f4cc7b84589e3c341fe12" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34573 } } peers { permanent_uuid: "7d4df22d924f4a5abe2959b079b21eb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43555 } }
I20260925 12:17:11.740515 1421 heartbeater.cc:503] Master 127.0.98.62:35803 was elected leader, sending a full tablet report...
I20260925 12:17:11.740515 1436 ts_tablet_manager.cc:1428] T 5fb495007c89451fb06cb6aa32df86f1 P 7d4df22d924f4a5abe2959b079b21eb6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 12:17:11.740638 1439 tablet_bootstrap.cc:654] T 5fb495007c89451fb06cb6aa32df86f1 P 47c0e0b4f03f4cc7b84589e3c341fe12: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:11.741139 1439 log.cc:824] T 5fb495007c89451fb06cb6aa32df86f1 P 47c0e0b4f03f4cc7b84589e3c341fe12: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:11.741351 1435 raft_consensus.cc:348] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b627538c484436288732d9333e425fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35071 } } peers { permanent_uuid: "47c0e0b4f03f4cc7b84589e3c341fe12" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34573 } } peers { permanent_uuid: "7d4df22d924f4a5abe2959b079b21eb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43555 } }
I20260925 12:17:11.741892 1435 raft_consensus.cc:374] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:11.741941 1435 raft_consensus.cc:729] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b627538c484436288732d9333e425fd, State: Initialized, Role: FOLLOWER
I20260925 12:17:11.742049 1435 consensus_queue.cc:260] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b627538c484436288732d9333e425fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35071 } } peers { permanent_uuid: "47c0e0b4f03f4cc7b84589e3c341fe12" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34573 } } peers { permanent_uuid: "7d4df22d924f4a5abe2959b079b21eb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43555 } }
I20260925 12:17:11.742326 1439 tablet_bootstrap.cc:492] T 5fb495007c89451fb06cb6aa32df86f1 P 47c0e0b4f03f4cc7b84589e3c341fe12: No bootstrap required, opened a new log
I20260925 12:17:11.742307 1435 ts_tablet_manager.cc:1428] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:17:11.742367 1287 heartbeater.cc:503] Master 127.0.98.62:35803 was elected leader, sending a full tablet report...
I20260925 12:17:11.742404 1439 ts_tablet_manager.cc:1397] T 5fb495007c89451fb06cb6aa32df86f1 P 47c0e0b4f03f4cc7b84589e3c341fe12: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:17:11.744022 1439 raft_consensus.cc:348] T 5fb495007c89451fb06cb6aa32df86f1 P 47c0e0b4f03f4cc7b84589e3c341fe12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b627538c484436288732d9333e425fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35071 } } peers { permanent_uuid: "47c0e0b4f03f4cc7b84589e3c341fe12" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34573 } } peers { permanent_uuid: "7d4df22d924f4a5abe2959b079b21eb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43555 } }
I20260925 12:17:11.744151 1439 raft_consensus.cc:374] T 5fb495007c89451fb06cb6aa32df86f1 P 47c0e0b4f03f4cc7b84589e3c341fe12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:11.744174 1439 raft_consensus.cc:729] T 5fb495007c89451fb06cb6aa32df86f1 P 47c0e0b4f03f4cc7b84589e3c341fe12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47c0e0b4f03f4cc7b84589e3c341fe12, State: Initialized, Role: FOLLOWER
I20260925 12:17:11.744297 1439 consensus_queue.cc:260] T 5fb495007c89451fb06cb6aa32df86f1 P 47c0e0b4f03f4cc7b84589e3c341fe12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b627538c484436288732d9333e425fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35071 } } peers { permanent_uuid: "47c0e0b4f03f4cc7b84589e3c341fe12" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34573 } } peers { permanent_uuid: "7d4df22d924f4a5abe2959b079b21eb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43555 } }
I20260925 12:17:11.744505 1439 ts_tablet_manager.cc:1428] T 5fb495007c89451fb06cb6aa32df86f1 P 47c0e0b4f03f4cc7b84589e3c341fe12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:17:11.744658 1153 heartbeater.cc:503] Master 127.0.98.62:35803 was elected leader, sending a full tablet report...
I20260925 12:17:11.764384 1442 raft_consensus.cc:482] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:17:11.764463 1442 raft_consensus.cc:504] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b627538c484436288732d9333e425fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35071 } } peers { permanent_uuid: "47c0e0b4f03f4cc7b84589e3c341fe12" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34573 } } peers { permanent_uuid: "7d4df22d924f4a5abe2959b079b21eb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43555 } }
I20260925 12:17:11.764753 1442 leader_election.cc:288] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47c0e0b4f03f4cc7b84589e3c341fe12 (127.0.98.1:34573), 7d4df22d924f4a5abe2959b079b21eb6 (127.0.98.3:43555)
I20260925 12:17:11.768347 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.765040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40115 (local address 127.0.98.1:34573)
0925 12:17:11.765075 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:11.765077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:11.765176 (+ 99us) server_negotiation.cc:407] Connection header received
0925 12:17:11.765202 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:11.765203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:11.765213 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:11.765258 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:11.765365 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.766025 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.767445 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.768010 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.768081 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.768090 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:11.768257 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 12:17:11.768290 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:17:11.768347 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.764967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60187 (local address 127.0.98.3:43555)
0925 12:17:11.765006 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:11.765008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:11.765017 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:17:11.765082 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:11.765083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:11.765094 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:11.765145 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:11.765319 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.766024 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.767445 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.768010 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.768081 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.768090 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:11.768256 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 12:17:11.768290 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:17:11.768364 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.764921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43555 (local address 127.0.98.2:60187)
0925 12:17:11.764962 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:11.764973 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:11.765011 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:11.765179 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:11.765180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:11.765181 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:11.765240 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:11.765242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.766107 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.766108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:11.767341 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:11.767343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.768095 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.768096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:11.768176 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.768178 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:17:11.768230 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 12:17:11.768280 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":50}
I20260925 12:17:11.768370 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.764921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34573 (local address 127.0.98.2:40115)
0925 12:17:11.765107 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:11.765115 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:11.765163 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:11.765262 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:11.765263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:11.765264 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:11.765317 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:11.765318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.766107 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.766108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:11.767341 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:11.767343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.768095 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.768096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:11.768169 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.768173 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:17:11.768230 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 12:17:11.768280 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":121,"thread_start_us":76,"threads_started":1}
I20260925 12:17:11.768721 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb495007c89451fb06cb6aa32df86f1" candidate_uuid: "2b627538c484436288732d9333e425fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4df22d924f4a5abe2959b079b21eb6" is_pre_election: true
I20260925 12:17:11.768815 1375 raft_consensus.cc:2454] T 5fb495007c89451fb06cb6aa32df86f1 P 7d4df22d924f4a5abe2959b079b21eb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b627538c484436288732d9333e425fd in term 0.
I20260925 12:17:11.768827 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb495007c89451fb06cb6aa32df86f1" candidate_uuid: "2b627538c484436288732d9333e425fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47c0e0b4f03f4cc7b84589e3c341fe12" is_pre_election: true
I20260925 12:17:11.768950 1107 raft_consensus.cc:2454] T 5fb495007c89451fb06cb6aa32df86f1 P 47c0e0b4f03f4cc7b84589e3c341fe12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b627538c484436288732d9333e425fd in term 0.
I20260925 12:17:11.769054 1177 leader_election.cc:302] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b627538c484436288732d9333e425fd, 7d4df22d924f4a5abe2959b079b21eb6; no voters:
I20260925 12:17:11.769191 1442 raft_consensus.cc:2781] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:17:11.769235 1442 raft_consensus.cc:482] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:17:11.769250 1442 raft_consensus.cc:3037] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:11.769798 1442 raft_consensus.cc:504] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b627538c484436288732d9333e425fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35071 } } peers { permanent_uuid: "47c0e0b4f03f4cc7b84589e3c341fe12" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34573 } } peers { permanent_uuid: "7d4df22d924f4a5abe2959b079b21eb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43555 } }
I20260925 12:17:11.769925 1442 leader_election.cc:288] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd [CANDIDATE]: Term 1 election: Requested vote from peers 47c0e0b4f03f4cc7b84589e3c341fe12 (127.0.98.1:34573), 7d4df22d924f4a5abe2959b079b21eb6 (127.0.98.3:43555)
I20260925 12:17:11.770105 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb495007c89451fb06cb6aa32df86f1" candidate_uuid: "2b627538c484436288732d9333e425fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47c0e0b4f03f4cc7b84589e3c341fe12"
I20260925 12:17:11.770174 1107 raft_consensus.cc:3037] T 5fb495007c89451fb06cb6aa32df86f1 P 47c0e0b4f03f4cc7b84589e3c341fe12 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:11.770136 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb495007c89451fb06cb6aa32df86f1" candidate_uuid: "2b627538c484436288732d9333e425fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4df22d924f4a5abe2959b079b21eb6"
I20260925 12:17:11.770236 1375 raft_consensus.cc:3037] T 5fb495007c89451fb06cb6aa32df86f1 P 7d4df22d924f4a5abe2959b079b21eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:11.770826 1107 raft_consensus.cc:2454] T 5fb495007c89451fb06cb6aa32df86f1 P 47c0e0b4f03f4cc7b84589e3c341fe12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b627538c484436288732d9333e425fd in term 1.
I20260925 12:17:11.770923 1375 raft_consensus.cc:2454] T 5fb495007c89451fb06cb6aa32df86f1 P 7d4df22d924f4a5abe2959b079b21eb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b627538c484436288732d9333e425fd in term 1.
I20260925 12:17:11.771027 1174 leader_election.cc:302] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b627538c484436288732d9333e425fd, 47c0e0b4f03f4cc7b84589e3c341fe12; no voters:
I20260925 12:17:11.771162 1442 raft_consensus.cc:2781] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:11.771289 1442 raft_consensus.cc:686] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd [term 1 LEADER]: Becoming Leader. State: Replica: 2b627538c484436288732d9333e425fd, State: Running, Role: LEADER
I20260925 12:17:11.771379 1442 consensus_queue.cc:237] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b627538c484436288732d9333e425fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35071 } } peers { permanent_uuid: "47c0e0b4f03f4cc7b84589e3c341fe12" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34573 } } peers { permanent_uuid: "7d4df22d924f4a5abe2959b079b21eb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43555 } }
I20260925 12:17:11.772089 962 catalog_manager.cc:5495] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b627538c484436288732d9333e425fd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2b627538c484436288732d9333e425fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b627538c484436288732d9333e425fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47c0e0b4f03f4cc7b84589e3c341fe12" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d4df22d924f4a5abe2959b079b21eb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43555 } health_report { overall_health: UNKNOWN } } }
Sep 25 12:17:11 dist-test-slave-clmx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:17:11.792411 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.785228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.0.98.1:34573)
0925 12:17:11.785262 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:11.785264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:11.785385 (+ 121us) server_negotiation.cc:407] Connection header received
0925 12:17:11.785458 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:11.785459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:11.785484 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:11.785546 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:11.786108 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.786545 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.787277 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.787457 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.787550 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.789708 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:11.789715 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:11.789716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:11.789723 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:11.791197 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:11.791571 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:11.791573 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:11.791574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:11.791682 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:11.791894 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:11.791895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:11.791896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:11.792032 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:11.792174 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:11.792249 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 12:17:11.792319 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 12:17:11.796063 1288 tablet.cc:2369] T 5fb495007c89451fb06cb6aa32df86f1 P 2b627538c484436288732d9333e425fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 12:17:11 dist-test-slave-clmx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338631, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 12:17:11 dist-test-slave-clmx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338631, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:17:11.814077 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.807555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.0.98.1:34573)
0925 12:17:11.807633 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:11.807635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:11.807747 (+ 112us) server_negotiation.cc:407] Connection header received
0925 12:17:11.807821 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:11.807822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:11.807831 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:11.807905 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:11.808411 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.808835 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.809530 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.809711 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.809822 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.811819 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:11.811821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:11.811821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:11.811828 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:11.812942 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:11.813297 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:11.813298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:11.813299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:11.813353 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:11.813563 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:11.813563 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:11.813564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:11.813654 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:11.813764 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:11.813915 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 12:17:11.813985 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 12:17:11 dist-test-slave-clmx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 12:17:11 dist-test-slave-clmx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:17:11.833806 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:11.827489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.0.98.1:34573)
0925 12:17:11.827545 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:11.827547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:11.827706 (+ 159us) server_negotiation.cc:407] Connection header received
0925 12:17:11.827768 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:11.827769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:11.827776 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:11.827819 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:11.828247 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.828629 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.829248 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:11.829429 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:11.829478 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:11.831653 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:11.831655 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:11.831655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:11.831667 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:11.832745 (+ 1078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:11.833088 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:11.833089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:11.833090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:11.833141 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:11.833355 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:11.833356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:11.833356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:11.833445 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:11.833558 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:11.833650 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 12:17:11.833716 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 12:17:11.834158 1067 tablet_service.cc:2908] Rejecting scan request for tablet 5fb495007c89451fb06cb6aa32df86f1: Uninitialized: clean time has not yet been initialized
I20260925 12:17:11.834970 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 1024
I20260925 12:17:11.839526 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 1158
I20260925 12:17:11.843791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 1292
I20260925 12:17:11.847885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 930
2026-09-25T12:17:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1888 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:17:11 dist-test-slave-clmx krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:17:11 dist-test-slave-clmx krb5kdc[1459](info): set up 2 sockets
Sep 25 12:17:11 dist-test-slave-clmx krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260925 12:17:12.842912 392 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:17:12 dist-test-slave-clmx krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:17:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:17:12.954308 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38751
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:13.031205 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:13.031445 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:13.031479 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:13.032656 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:13.032735 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 12:17:13.032756 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:13.032778 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:13.032800 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:13.032817 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:17:13.034178 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38751
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:13.034425 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:13.034632 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:13.039146 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:13.039155 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:13.039173 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:13.039335 1472 server_base.cc:1034] running on GCE node
I20260925 12:17:13.039582 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:13.039892 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:13.041059 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790338633041043 us; error 47 us; skew 500 ppm
Sep 25 12:17:13 dist-test-slave-clmx krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:13.043447 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:17:13.043879 1472 webserver.cc:460] Webserver started at http://127.0.98.62:41009/ using document root <none> and password file <none>
I20260925 12:17:13.044039 1472 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:13.044077 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:13.044167 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:13.045104 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "8c985f96aa294557932e7b01e8687deb"
format_stamp: "Formatted at 2026-09-25 12:17:13 on dist-test-slave-clmx"
server_key: "010665e7334823bf9971f60fbf6598fa"
server_key_iv: "95a4e6cbb4c7a47babd3b3a1773f5e44"
server_key_version: "encryptionkey@0"
I20260925 12:17:13.045392 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "8c985f96aa294557932e7b01e8687deb"
format_stamp: "Formatted at 2026-09-25 12:17:13 on dist-test-slave-clmx"
server_key: "010665e7334823bf9971f60fbf6598fa"
server_key_iv: "95a4e6cbb4c7a47babd3b3a1773f5e44"
server_key_version: "encryptionkey@0"
I20260925 12:17:13.046561 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:17:13.047185 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:13.047376 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:13.047441 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "8c985f96aa294557932e7b01e8687deb"
format_stamp: "Formatted at 2026-09-25 12:17:13 on dist-test-slave-clmx"
server_key: "010665e7334823bf9971f60fbf6598fa"
server_key_iv: "95a4e6cbb4c7a47babd3b3a1773f5e44"
server_key_version: "encryptionkey@0"
I20260925 12:17:13.047528 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:13.057526 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:13.059513 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:13.059608 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:13.064013 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38751
I20260925 12:17:13.064075 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38751 every 8 connection(s)
I20260925 12:17:13.064373 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:13.064947 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:13.066978 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb: Bootstrap starting.
I20260925 12:17:13.067601 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:13.067951 1550 log.cc:824] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:13.068550 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb: No bootstrap required, opened a new log
I20260925 12:17:13.068804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 1472
I20260925 12:17:13.068893 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:17:13.069046 392 external_mini_cluster.cc:1428] Setting key 2b2c4fcd19620995b35bdc25954fb2d0
I20260925 12:17:13.070878 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c985f96aa294557932e7b01e8687deb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38751 } }
I20260925 12:17:13.071031 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:13.071075 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c985f96aa294557932e7b01e8687deb, State: Initialized, Role: FOLLOWER
I20260925 12:17:13.071213 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c985f96aa294557932e7b01e8687deb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38751 } }
I20260925 12:17:13.071300 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:13.071334 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:13.071385 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:13.072021 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c985f96aa294557932e7b01e8687deb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38751 } }
I20260925 12:17:13.072121 1550 leader_election.cc:302] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c985f96aa294557932e7b01e8687deb; no voters:
I20260925 12:17:13.072293 1550 leader_election.cc:288] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:17:13.072371 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:13.072482 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb [term 1 LEADER]: Becoming Leader. State: Replica: 8c985f96aa294557932e7b01e8687deb, State: Running, Role: LEADER
I20260925 12:17:13.072549 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:13.072556 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c985f96aa294557932e7b01e8687deb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38751 } }
I20260925 12:17:13.072854 1555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c985f96aa294557932e7b01e8687deb. Latest consensus state: current_term: 1 leader_uuid: "8c985f96aa294557932e7b01e8687deb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c985f96aa294557932e7b01e8687deb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38751 } } }
I20260925 12:17:13.072916 1555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:13.073030 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c985f96aa294557932e7b01e8687deb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c985f96aa294557932e7b01e8687deb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38751 } } }
I20260925 12:17:13.073081 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:13.073186 1560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:13.073657 1560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:17:13.075145 1560 catalog_manager.cc:1347] Generated new cluster ID: 25f38c2eaf794a8a9d587f9729bb5129
I20260925 12:17:13.075192 1560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:17:13 dist-test-slave-clmx krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:13.080650 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.069479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36454 (local address 127.0.98.62:38751)
0925 12:17:13.069617 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.069625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.069652 (+ 27us) server_negotiation.cc:407] Connection header received
0925 12:17:13.070875 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.070880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.070930 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.071071 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:13.071815 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.073067 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.074090 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.074554 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.074651 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.077254 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:13.077257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:13.077258 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:13.077267 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:13.078987 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.079336 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.079339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.079339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.079424 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.079571 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.079571 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.079571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.079815 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:13.079936 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.080347 (+ 411us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.080460 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":75,"threads_started":1}
I20260925 12:17:13.103749 1560 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:17:13.104453 1560 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:13.113569 1560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb: Generated new TSK 0
I20260925 12:17:13.113732 1560 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:17:13.139391 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38751
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:13.214453 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:13.214672 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:13.214709 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:13.215813 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:13.215880 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:13.215917 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 12:17:13.215950 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:17:13.217271 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:13.217505 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:13.217710 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:13.222570 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:13.222576 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:13.222630 1576 server_base.cc:1034] running on GCE node
W20260925 12:17:13.222645 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:13.222939 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:13.223167 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:13.224331 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790338633224313 us; error 29 us; skew 500 ppm
Sep 25 12:17:13 dist-test-slave-clmx krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:13.226558 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:17:13.226948 1576 webserver.cc:460] Webserver started at http://127.0.98.1:33597/ using document root <none> and password file <none>
I20260925 12:17:13.227152 1576 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:13.227222 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:13.227368 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:13.228398 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "2d925a59d87a4ba4a93b5ac7b459570e"
format_stamp: "Formatted at 2026-09-25 12:17:13 on dist-test-slave-clmx"
server_key: "2c11981368791b324f9632e287672ce9"
server_key_iv: "41249a6c94d8078d605bcdf0fa183bf6"
server_key_version: "encryptionkey@0"
I20260925 12:17:13.228753 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2d925a59d87a4ba4a93b5ac7b459570e"
format_stamp: "Formatted at 2026-09-25 12:17:13 on dist-test-slave-clmx"
server_key: "2c11981368791b324f9632e287672ce9"
server_key_iv: "41249a6c94d8078d605bcdf0fa183bf6"
server_key_version: "encryptionkey@0"
I20260925 12:17:13.230014 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:17:13.230692 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:13.230870 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:17:13.230968 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "2d925a59d87a4ba4a93b5ac7b459570e"
format_stamp: "Formatted at 2026-09-25 12:17:13 on dist-test-slave-clmx"
server_key: "2c11981368791b324f9632e287672ce9"
server_key_iv: "41249a6c94d8078d605bcdf0fa183bf6"
server_key_version: "encryptionkey@0"
I20260925 12:17:13.231055 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:13.254987 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:13.256557 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:13.256736 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:13.257253 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:13.257316 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:13.257347 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:13.257375 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 12:17:13 dist-test-slave-clmx krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:13.264364 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45319
I20260925 12:17:13.264547 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45319 every 8 connection(s)
I20260925 12:17:13.264750 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:17:13.270438 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.257660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52685 (local address 127.0.98.62:38751)
0925 12:17:13.257743 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.257746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.257919 (+ 173us) server_negotiation.cc:407] Connection header received
0925 12:17:13.258328 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.258329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.258346 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.258422 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:13.259136 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.259977 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.260998 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.261375 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.261448 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.263929 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:13.263930 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:13.263930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:13.263938 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:13.268802 (+ 4864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.269197 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.269198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.269198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.269249 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.269456 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.269456 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.269457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.269567 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:13.269616 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.270279 (+ 663us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.270358 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 12:17:13.271493 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.257609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38751 (local address 127.0.98.1:52685)
0925 12:17:13.257815 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:13.257839 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:13.258234 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:13.258484 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:13.258487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:13.258766 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:13.259027 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.259032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.260093 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.260094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.260930 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.260932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.261484 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.261485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.261642 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.262123 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:13.262133 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:13.263854 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:13.268888 (+ 5034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:13.268893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:13.268895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:13.269125 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:13.269304 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:13.269304 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:13.269305 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:13.269400 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:13.269635 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:13.269650 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:13.269775 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 12:17:13.269858 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 12:17:13.269985 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":68,"threads_started":1}
I20260925 12:17:13.272162 1707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38751
I20260925 12:17:13.272245 1707 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:13.272411 1707 heartbeater.cc:511] Master 127.0.98.62:38751 requested a full tablet report, sending...
I20260925 12:17:13.272914 1504 ts_manager.cc:194] Registered new tserver with Master: 2d925a59d87a4ba4a93b5ac7b459570e (127.0.98.1:45319)
I20260925 12:17:13.273693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 1576
I20260925 12:17:13.273764 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:17:13.273787 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52685
I20260925 12:17:13.273960 392 external_mini_cluster.cc:1428] Setting key 063bb2394253311865bc18c8ad4d06c3
I20260925 12:17:13.279359 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.273237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44739 (local address 127.0.98.62:38751)
0925 12:17:13.273268 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.273270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.273464 (+ 194us) server_negotiation.cc:407] Connection header received
0925 12:17:13.273485 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.273486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.273496 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.273574 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:13.274449 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.274961 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.275821 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.275990 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.276044 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.276895 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:13.276896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:13.276896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:13.276902 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:13.278321 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.278714 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.278715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.278715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.278762 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.278975 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.278975 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.278975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.279054 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:13.279098 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.279262 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.279312 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 12:17:13.279475 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.273202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38751 (local address 127.0.98.1:44739)
0925 12:17:13.273372 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:13.273381 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:13.273437 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:13.274131 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:13.274132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:13.274329 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:13.274387 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.274389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.275059 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.275060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.275737 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.275740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.276066 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.276067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.276131 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.276491 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:13.276496 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:13.276813 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:13.278401 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:13.278401 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:13.278402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:13.278635 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:13.278844 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:13.278845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:13.278845 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:13.278912 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:13.279116 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:13.279117 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:13.279168 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 12:17:13.279248 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 12:17:13.279343 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":131,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:17:13.298597 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38751
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:13.373267 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:13.373581 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:13.373616 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:13.374684 1716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:13.374718 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:13.374737 1716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 12:17:13.374790 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:17:13.376098 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:13.376313 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:13.376518 1716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:13.381158 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:13.381143 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:13.381264 1716 server_base.cc:1034] running on GCE node
W20260925 12:17:13.381160 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:13.381534 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:13.381772 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:13.382906 1716 hybrid_clock.cc:648] HybridClock initialized: now 1790338633382883 us; error 38 us; skew 500 ppm
Sep 25 12:17:13 dist-test-slave-clmx krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:13.385207 1716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:17:13.385687 1716 webserver.cc:460] Webserver started at http://127.0.98.2:40499/ using document root <none> and password file <none>
I20260925 12:17:13.385918 1716 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:13.385988 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:13.386137 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:13.387084 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "6c670ff22cbb498fbce3efc7a9499d8d"
format_stamp: "Formatted at 2026-09-25 12:17:13 on dist-test-slave-clmx"
server_key: "79df041a24a4adbdfe7dc7dc72190b71"
server_key_iv: "2358149feabdbbe33c2759f6aedde3ef"
server_key_version: "encryptionkey@0"
I20260925 12:17:13.387418 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6c670ff22cbb498fbce3efc7a9499d8d"
format_stamp: "Formatted at 2026-09-25 12:17:13 on dist-test-slave-clmx"
server_key: "79df041a24a4adbdfe7dc7dc72190b71"
server_key_iv: "2358149feabdbbe33c2759f6aedde3ef"
server_key_version: "encryptionkey@0"
I20260925 12:17:13.388684 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:17:13.389397 1731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:13.389556 1716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:17:13.389654 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "6c670ff22cbb498fbce3efc7a9499d8d"
format_stamp: "Formatted at 2026-09-25 12:17:13 on dist-test-slave-clmx"
server_key: "79df041a24a4adbdfe7dc7dc72190b71"
server_key_iv: "2358149feabdbbe33c2759f6aedde3ef"
server_key_version: "encryptionkey@0"
I20260925 12:17:13.389739 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:13.399552 1716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:13.401716 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:13.401873 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:13.402398 1716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:13.402444 1716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:13.402475 1716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:13.402493 1716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 12:17:13 dist-test-slave-clmx krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:13.409550 1716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46003
I20260925 12:17:13.409914 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:17:13.409932 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46003 every 8 connection(s)
I20260925 12:17:13.411767 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.402800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41949 (local address 127.0.98.62:38751)
0925 12:17:13.402866 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.402869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.403026 (+ 157us) server_negotiation.cc:407] Connection header received
0925 12:17:13.403472 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.403473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.403508 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.403580 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:13.404308 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.404815 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.405854 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.406136 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.406233 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.409118 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:13.409119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:13.409120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:13.409127 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:13.410651 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.411006 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.411006 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.411007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.411054 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.411238 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.411238 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.411238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.411322 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:13.411366 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.411570 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.411682 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 12:17:13.411918 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.402768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38751 (local address 127.0.98.2:41949)
0925 12:17:13.402903 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:13.402937 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:13.403377 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:13.403703 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:13.403707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:13.403937 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:13.404166 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.404171 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.404937 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.404938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.405722 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.405724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.406242 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.406243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.406381 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.406886 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:13.406897 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:13.409046 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:13.410721 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:13.410724 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:13.410725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:13.410938 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:13.411111 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:13.411111 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:13.411111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:13.411198 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:13.411373 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:13.411376 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:13.411472 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 12:17:13.411556 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 12:17:13.411721 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20260925 12:17:13.412387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 1716
I20260925 12:17:13.412410 1847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38751
I20260925 12:17:13.412468 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:17:13.412479 1847 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:13.412644 392 external_mini_cluster.cc:1428] Setting key 53f52e300e8e8797d457edf65833215b
I20260925 12:17:13.412660 1847 heartbeater.cc:511] Master 127.0.98.62:38751 requested a full tablet report, sending...
I20260925 12:17:13.412986 1503 ts_manager.cc:194] Registered new tserver with Master: 6c670ff22cbb498fbce3efc7a9499d8d (127.0.98.2:46003)
I20260925 12:17:13.413511 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41949
I20260925 12:17:13.418659 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.413446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43727 (local address 127.0.98.62:38751)
0925 12:17:13.413481 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.413483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.413594 (+ 111us) server_negotiation.cc:407] Connection header received
0925 12:17:13.413712 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.413713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.413725 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.413775 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:13.414099 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.414710 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.415374 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.415557 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.415634 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.416481 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:13.416483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:13.416483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:13.416489 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:13.417635 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.417960 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.417961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.417961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.418008 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.418192 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.418193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.418194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.418322 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:13.418398 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.418524 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.418582 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:17:13.418666 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.413386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38751 (local address 127.0.98.2:43727)
0925 12:17:13.413534 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:13.413545 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:13.413617 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:13.413786 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:13.413786 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:13.413970 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:13.414017 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.414018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.414805 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.414806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.415282 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.415284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.415655 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.415656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.415713 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.416084 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:13.416087 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:13.416391 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:13.417740 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:13.417740 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:13.417741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:13.417890 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:13.418070 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:13.418071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:13.418071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:13.418119 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:13.418391 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:13.418392 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:13.418428 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 12:17:13.418488 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 12:17:13.418575 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:17:13.438087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38751
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:13.517212 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:13.517388 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:13.517410 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:13.518416 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:13.518448 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:13.518467 1856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 12:17:13.518484 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:17:13.519799 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:13.519959 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:13.520100 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:13.524798 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:13.524824 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:13.524848 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:13.525169 1856 server_base.cc:1034] running on GCE node
I20260925 12:17:13.525403 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:13.525656 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:13.526835 1856 hybrid_clock.cc:648] HybridClock initialized: now 1790338633526820 us; error 26 us; skew 500 ppm
Sep 25 12:17:13 dist-test-slave-clmx krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:13.529153 1856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:17:13.529654 1856 webserver.cc:460] Webserver started at http://127.0.98.3:35497/ using document root <none> and password file <none>
I20260925 12:17:13.529876 1856 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:13.529949 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:13.530113 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:13.531095 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "424446f817934078b5757ba12282c422"
format_stamp: "Formatted at 2026-09-25 12:17:13 on dist-test-slave-clmx"
server_key: "a1c8ac6f4d9a5c283f4e064270f97c98"
server_key_iv: "efcce668e972a4f1f19736d303c1a785"
server_key_version: "encryptionkey@0"
I20260925 12:17:13.531445 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "424446f817934078b5757ba12282c422"
format_stamp: "Formatted at 2026-09-25 12:17:13 on dist-test-slave-clmx"
server_key: "a1c8ac6f4d9a5c283f4e064270f97c98"
server_key_iv: "efcce668e972a4f1f19736d303c1a785"
server_key_version: "encryptionkey@0"
I20260925 12:17:13.532730 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:13.533418 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:13.533604 1856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:13.533727 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "424446f817934078b5757ba12282c422"
format_stamp: "Formatted at 2026-09-25 12:17:13 on dist-test-slave-clmx"
server_key: "a1c8ac6f4d9a5c283f4e064270f97c98"
server_key_iv: "efcce668e972a4f1f19736d303c1a785"
server_key_version: "encryptionkey@0"
I20260925 12:17:13.533826 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:13.544390 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:13.546170 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:13.546329 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:13.546902 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:13.546983 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:13.547017 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:13.547046 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 12:17:13 dist-test-slave-clmx krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:13.554705 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41375
I20260925 12:17:13.554780 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41375 every 8 connection(s)
I20260925 12:17:13.555121 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:17:13.557394 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.547279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47977 (local address 127.0.98.62:38751)
0925 12:17:13.547347 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.547349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.547592 (+ 243us) server_negotiation.cc:407] Connection header received
0925 12:17:13.548356 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.548357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.548367 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.548406 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:13.549274 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.550028 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.551297 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.551594 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.551689 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.554609 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:13.554611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:13.554611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:13.554619 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:13.556110 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.556501 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.556502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.556502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.556554 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.556816 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.556817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.556817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.556924 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:13.556998 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.557244 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.557314 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 12:17:13.557576 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.547259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38751 (local address 127.0.98.3:47977)
0925 12:17:13.547449 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:13.547492 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:13.548214 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:13.548476 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:13.548480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:13.548861 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:13.549151 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.549158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.550145 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.550147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.551188 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.551190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.551700 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.551701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.551861 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.552436 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:13.552453 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:13.554469 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:13.556206 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:13.556212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:13.556214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:13.556407 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:13.556630 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:13.556631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:13.556631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:13.556734 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:13.556991 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:13.556996 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:13.557119 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 12:17:13.557222 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 12:17:13.557354 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
I20260925 12:17:13.558097 1987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38751
I20260925 12:17:13.558292 1987 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:13.558461 1987 heartbeater.cc:511] Master 127.0.98.62:38751 requested a full tablet report, sending...
I20260925 12:17:13.558821 1504 ts_manager.cc:194] Registered new tserver with Master: 424446f817934078b5757ba12282c422 (127.0.98.3:41375)
I20260925 12:17:13.559226 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47977
I20260925 12:17:13.562021 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 1856
I20260925 12:17:13.562131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:17:13.562294 392 external_mini_cluster.cc:1428] Setting key 8be2864567b0760215642c685ad356b2
I20260925 12:17:13.563350 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:17:13.564888 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.559003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48687 (local address 127.0.98.62:38751)
0925 12:17:13.559350 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.559352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.559455 (+ 103us) server_negotiation.cc:407] Connection header received
0925 12:17:13.559512 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.559513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.559521 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.559569 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:13.559923 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.560367 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.561050 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.561221 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.561269 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.562042 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:13.562043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:13.562044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:13.562050 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:13.563749 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.564094 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.564095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.564095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.564153 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.564397 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.564398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.564398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.564503 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:13.564580 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.564737 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.564804 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329}
I20260925 12:17:13.564905 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.558939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38751 (local address 127.0.98.3:48687)
0925 12:17:13.559408 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:13.559417 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:13.559467 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:13.559576 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:13.559576 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:13.559795 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:13.559843 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.559845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.560465 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.560466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.560976 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.560978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.561317 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.561317 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.561381 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.561714 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:13.561717 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:13.561956 (+ 239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:13.563841 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:13.563842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:13.563842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:13.564003 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:13.564261 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:13.564262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:13.564262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:13.564317 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:13.564573 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:13.564574 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:13.564621 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 12:17:13.564714 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 12:17:13.564804 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":60,"threads_started":1}
Sep 25 12:17:13 dist-test-slave-clmx krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 12:17:13 dist-test-slave-clmx krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:13.595399 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.588209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36462 (local address 127.0.98.62:38751)
0925 12:17:13.588276 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.588278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.588285 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:17:13.588305 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.588306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.588315 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.588399 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:13.589234 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.589711 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.590506 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.590693 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.590907 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.593064 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:13.593065 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:13.593065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:13.593071 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:13.594475 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.594772 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.594772 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.594773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.594818 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.595028 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.595028 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.595028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.595115 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:13.595183 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.595268 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.595316 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 12:17:13.596921 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:17:13.598400 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:17:13.605311 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.601013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38884 (local address 127.0.98.1:45319)
0925 12:17:13.601179 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.601183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.601194 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:17:13.601235 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.601238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.601254 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.601341 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:13.601539 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.602627 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.604025 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.604763 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.604836 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.604909 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.605172 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.605214 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260925 12:17:13.605295 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.600695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46003 (local address 127.0.0.1:48080)
0925 12:17:13.600837 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:13.600877 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:13.600944 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:13.601626 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:13.601627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:13.601630 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:13.601687 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.601689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.602559 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.602560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.603933 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.603936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.604947 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.604948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.605051 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.605059 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 12:17:13.605147 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 12:17:13.605206 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":97,"thread_start_us":46,"threads_started":1}
I20260925 12:17:13.605296 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.600900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45319 (local address 127.0.0.1:38884)
0925 12:17:13.600974 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:13.600984 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:13.601032 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:13.601378 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:13.601381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:13.601383 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:13.601446 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.601447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.602724 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.602724 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.603933 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.603936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.604949 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.604949 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.605047 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.605059 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 12:17:13.605137 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 12:17:13.605206 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":8,"thread_start_us":34,"threads_started":1}
I20260925 12:17:13.605638 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.600770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48080 (local address 127.0.98.2:46003)
0925 12:17:13.601448 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.601452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.601462 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:17:13.601498 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.601499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.601514 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.601762 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:13.601821 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.602454 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.604024 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.604763 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.604836 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.604909 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.605499 (+ 590us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.605542 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":83,"threads_started":1}
I20260925 12:17:13.606171 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.600717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41375 (local address 127.0.0.1:52568)
0925 12:17:13.600975 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:13.600985 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:13.601032 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:13.601378 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:13.601381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:13.601385 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:13.601446 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.601447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.604032 (+ 2585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.604033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.604714 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.604715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.605925 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.605926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.605995 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.605999 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:17:13.606056 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 12:17:13.606103 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":3,"thread_start_us":47,"threads_started":1}
I20260925 12:17:13.606172 1641 tablet_service.cc:1467] Processing CreateTablet for tablet c63d20b1d7114e6fa591b5086e11690c (DEFAULT_TABLE table=test-table [id=bc78918640c5445090ae7f0fe7f09b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:13.606190 1781 tablet_service.cc:1467] Processing CreateTablet for tablet c63d20b1d7114e6fa591b5086e11690c (DEFAULT_TABLE table=test-table [id=bc78918640c5445090ae7f0fe7f09b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:13.606436 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.600777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52568 (local address 127.0.98.3:41375)
0925 12:17:13.601222 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.601226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.601234 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:13.601264 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.601265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.601281 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.601352 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:13.601538 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.602642 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.604808 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.605760 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.605825 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.605894 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.606095 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.606145 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":85,"threads_started":1}
I20260925 12:17:13.606487 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63d20b1d7114e6fa591b5086e11690c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:13.606510 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63d20b1d7114e6fa591b5086e11690c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:13.607159 1905 tablet_service.cc:1467] Processing CreateTablet for tablet c63d20b1d7114e6fa591b5086e11690c (DEFAULT_TABLE table=test-table [id=bc78918640c5445090ae7f0fe7f09b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:13.607414 1905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63d20b1d7114e6fa591b5086e11690c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:13.609254 2007 tablet_bootstrap.cc:492] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d: Bootstrap starting.
I20260925 12:17:13.609777 2007 tablet_bootstrap.cc:654] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:13.609978 2007 log.cc:824] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:13.610582 2009 tablet_bootstrap.cc:492] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e: Bootstrap starting.
I20260925 12:17:13.610618 2007 tablet_bootstrap.cc:492] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d: No bootstrap required, opened a new log
I20260925 12:17:13.610666 2007 ts_tablet_manager.cc:1397] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:13.610939 2010 tablet_bootstrap.cc:492] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422: Bootstrap starting.
I20260925 12:17:13.611155 2009 tablet_bootstrap.cc:654] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:13.611459 2009 log.cc:824] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:13.611481 2010 tablet_bootstrap.cc:654] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:13.611886 2010 log.cc:824] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:13.612229 2009 tablet_bootstrap.cc:492] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e: No bootstrap required, opened a new log
I20260925 12:17:13.612291 2009 ts_tablet_manager.cc:1397] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 12:17:13.612742 2010 tablet_bootstrap.cc:492] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422: No bootstrap required, opened a new log
I20260925 12:17:13.612807 2010 ts_tablet_manager.cc:1397] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 12:17:13.612908 2007 raft_consensus.cc:348] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } } peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } }
I20260925 12:17:13.613066 2007 raft_consensus.cc:374] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:13.613119 2007 raft_consensus.cc:729] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c670ff22cbb498fbce3efc7a9499d8d, State: Initialized, Role: FOLLOWER
I20260925 12:17:13.613240 2007 consensus_queue.cc:260] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } } peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } }
I20260925 12:17:13.613480 2007 ts_tablet_manager.cc:1428] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:17:13.613610 1847 heartbeater.cc:503] Master 127.0.98.62:38751 was elected leader, sending a full tablet report...
I20260925 12:17:13.614209 2009 raft_consensus.cc:348] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } } peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } }
I20260925 12:17:13.614400 2009 raft_consensus.cc:374] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:13.614444 2009 raft_consensus.cc:729] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d925a59d87a4ba4a93b5ac7b459570e, State: Initialized, Role: FOLLOWER
I20260925 12:17:13.614563 2009 consensus_queue.cc:260] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } } peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } }
I20260925 12:17:13.614840 2009 ts_tablet_manager.cc:1428] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 12:17:13.614914 1707 heartbeater.cc:503] Master 127.0.98.62:38751 was elected leader, sending a full tablet report...
I20260925 12:17:13.615092 2010 raft_consensus.cc:348] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } } peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } }
I20260925 12:17:13.615240 2010 raft_consensus.cc:374] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:13.615280 2010 raft_consensus.cc:729] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 424446f817934078b5757ba12282c422, State: Initialized, Role: FOLLOWER
I20260925 12:17:13.615379 2010 consensus_queue.cc:260] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } } peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } }
I20260925 12:17:13.615691 2010 ts_tablet_manager.cc:1428] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 12:17:13.615839 1987 heartbeater.cc:503] Master 127.0.98.62:38751 was elected leader, sending a full tablet report...
W20260925 12:17:13.660508 1848 tablet.cc:2369] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:17:13.664754 2015 raft_consensus.cc:482] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:17:13.664844 2015 raft_consensus.cc:504] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } } peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } }
I20260925 12:17:13.665150 2015 leader_election.cc:288] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c670ff22cbb498fbce3efc7a9499d8d (127.0.98.2:46003), 2d925a59d87a4ba4a93b5ac7b459570e (127.0.98.1:45319)
I20260925 12:17:13.667878 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.665370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55995 (local address 127.0.98.2:46003)
0925 12:17:13.665412 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.665414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.665422 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:13.665449 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.665450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.665460 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.665558 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:13.665718 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.666207 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.667133 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.667488 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.667585 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.667595 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.667772 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.667810 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 12:17:13.667898 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.665302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46003 (local address 127.0.98.3:55995)
0925 12:17:13.665339 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:13.665353 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:13.665397 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:13.665564 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:13.665565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:13.665566 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:13.665645 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.665646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.666296 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.666297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.667030 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.667031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.667593 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.667594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.667691 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.667693 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:17:13.667768 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 12:17:13.667806 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 12:17:13.668157 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.665432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44959 (local address 127.0.98.1:45319)
0925 12:17:13.665486 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.665489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.665587 (+ 98us) server_negotiation.cc:407] Connection header received
0925 12:17:13.665680 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.665681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.665694 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.665783 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:13.665955 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.666398 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.667307 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.667723 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.667813 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.667822 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.668035 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.668084 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:17:13.668237 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63d20b1d7114e6fa591b5086e11690c" candidate_uuid: "424446f817934078b5757ba12282c422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" is_pre_election: true
I20260925 12:17:13.668370 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.665330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45319 (local address 127.0.98.3:44959)
0925 12:17:13.665509 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:13.665517 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:13.665612 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:13.665785 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:13.665786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:13.665787 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:13.665877 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.665879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.666496 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.666497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.667212 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.667214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.667814 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.667815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.667890 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.667893 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:13.668234 (+ 341us) client_negotiation.cc:240] Negotiation successful
0925 12:17:13.668280 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":94,"threads_started":1}
I20260925 12:17:13.668381 1801 raft_consensus.cc:2454] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 424446f817934078b5757ba12282c422 in term 0.
I20260925 12:17:13.668637 1873 leader_election.cc:302] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 424446f817934078b5757ba12282c422, 6c670ff22cbb498fbce3efc7a9499d8d; no voters:
I20260925 12:17:13.668686 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63d20b1d7114e6fa591b5086e11690c" candidate_uuid: "424446f817934078b5757ba12282c422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" is_pre_election: true
I20260925 12:17:13.668773 2015 raft_consensus.cc:2781] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:17:13.668824 2015 raft_consensus.cc:482] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:17:13.668834 1661 raft_consensus.cc:2454] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 424446f817934078b5757ba12282c422 in term 0.
I20260925 12:17:13.668856 2015 raft_consensus.cc:3037] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:13.669484 2015 raft_consensus.cc:504] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } } peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } }
I20260925 12:17:13.669618 2015 leader_election.cc:288] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [CANDIDATE]: Term 1 election: Requested vote from peers 6c670ff22cbb498fbce3efc7a9499d8d (127.0.98.2:46003), 2d925a59d87a4ba4a93b5ac7b459570e (127.0.98.1:45319)
I20260925 12:17:13.669724 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63d20b1d7114e6fa591b5086e11690c" candidate_uuid: "424446f817934078b5757ba12282c422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d925a59d87a4ba4a93b5ac7b459570e"
I20260925 12:17:13.669737 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63d20b1d7114e6fa591b5086e11690c" candidate_uuid: "424446f817934078b5757ba12282c422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c670ff22cbb498fbce3efc7a9499d8d"
I20260925 12:17:13.669792 1661 raft_consensus.cc:3037] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:13.669795 1801 raft_consensus.cc:3037] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:13.670379 1801 raft_consensus.cc:2454] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 424446f817934078b5757ba12282c422 in term 1.
I20260925 12:17:13.670501 1661 raft_consensus.cc:2454] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 424446f817934078b5757ba12282c422 in term 1.
I20260925 12:17:13.670594 1873 leader_election.cc:302] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 424446f817934078b5757ba12282c422, 6c670ff22cbb498fbce3efc7a9499d8d; no voters:
I20260925 12:17:13.670697 2015 raft_consensus.cc:2781] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:13.670853 2015 raft_consensus.cc:686] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [term 1 LEADER]: Becoming Leader. State: Replica: 424446f817934078b5757ba12282c422, State: Running, Role: LEADER
I20260925 12:17:13.670975 2015 consensus_queue.cc:237] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } } peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } }
I20260925 12:17:13.671732 1504 catalog_manager.cc:5495] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 reported cstate change: term changed from 0 to 1, leader changed from <none> to 424446f817934078b5757ba12282c422 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "424446f817934078b5757ba12282c422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } health_report { overall_health: UNKNOWN } } }
I20260925 12:17:13.681072 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.678748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52580 (local address 127.0.98.3:41375)
0925 12:17:13.678892 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.678894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.678899 (+ 5us) server_negotiation.cc:407] Connection header received
0925 12:17:13.678918 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.678918 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.678930 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.678971 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:13.679136 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.679529 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.680295 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.680447 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.680499 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.680698 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:13.680775 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:13.680831 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.681012 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.681027 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260925 12:17:13.683146 1801 raft_consensus.cc:1261] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d [term 1 FOLLOWER]: Refusing update from remote peer 424446f817934078b5757ba12282c422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:13.683146 1661 raft_consensus.cc:1261] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e [term 1 FOLLOWER]: Refusing update from remote peer 424446f817934078b5757ba12282c422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:13.683452 2017 consensus_queue.cc:1035] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:13.683609 2015 consensus_queue.cc:1035] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:13.685835 2021 mvcc.cc:205] Tried to move back new op lower bound from 7333227043563335680 to 7333227043516981248. Current Snapshot: MvccSnapshot[applied={T|T < 7333227043563335680}]
I20260925 12:17:13.686161 2022 mvcc.cc:205] Tried to move back new op lower bound from 7333227043563335680 to 7333227043516981248. Current Snapshot: MvccSnapshot[applied={T|T < 7333227043563335680}]
I20260925 12:17:13.689778 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:13.689850 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:13.690572 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb: Sending DeleteTablet for 3 replicas of tablet c63d20b1d7114e6fa591b5086e11690c
I20260925 12:17:13.690932 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet c63d20b1d7114e6fa591b5086e11690c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38884
I20260925 12:17:13.691015 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet c63d20b1d7114e6fa591b5086e11690c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48080
I20260925 12:17:13.691082 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 12:17:13.691182 1905 tablet_service.cc:1514] Processing DeleteTablet for tablet c63d20b1d7114e6fa591b5086e11690c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52568
W20260925 12:17:13.691216 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:17:13.691413 2030 tablet_replica.cc:330] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e: stopping tablet replica
I20260925 12:17:13.691510 2030 raft_consensus.cc:2229] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:13.691661 2030 raft_consensus.cc:2258] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:13.691809 2031 tablet_replica.cc:330] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d: stopping tablet replica
I20260925 12:17:13.692217 2030 ts_tablet_manager.cc:1905] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:13.692461 1641 tablet_service.cc:1467] Processing CreateTablet for tablet aeb6290071ad4cd4a925e96ad543391f (DEFAULT_TABLE table=test-table [id=712ade676549433cb9f3b2c49dcd81e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:13.692597 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb6290071ad4cd4a925e96ad543391f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:13.693827 1781 tablet_service.cc:1467] Processing CreateTablet for tablet aeb6290071ad4cd4a925e96ad543391f (DEFAULT_TABLE table=test-table [id=712ade676549433cb9f3b2c49dcd81e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:13.693912 2009 tablet_bootstrap.cc:492] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e: Bootstrap starting.
I20260925 12:17:13.693925 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb6290071ad4cd4a925e96ad543391f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:13.694110 2031 raft_consensus.cc:2229] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:13.694206 2031 raft_consensus.cc:2258] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:13.694402 2032 tablet_replica.cc:330] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422: stopping tablet replica
I20260925 12:17:13.694483 2032 raft_consensus.cc:2229] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:17:13.694612 2032 raft_consensus.cc:2258] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:13.694737 1905 tablet_service.cc:1467] Processing CreateTablet for tablet aeb6290071ad4cd4a925e96ad543391f (DEFAULT_TABLE table=test-table [id=712ade676549433cb9f3b2c49dcd81e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:13.694832 1905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb6290071ad4cd4a925e96ad543391f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:13.694993 2032 ts_tablet_manager.cc:1905] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:13.695189 2009 tablet_bootstrap.cc:654] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:13.695771 2030 ts_tablet_manager.cc:1918] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:17:13.695889 2030 log.cc:1196] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-0/wal/wals/c63d20b1d7114e6fa591b5086e11690c
I20260925 12:17:13.696136 2010 tablet_bootstrap.cc:492] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422: Bootstrap starting.
I20260925 12:17:13.696249 2032 ts_tablet_manager.cc:1918] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:17:13.696305 2032 log.cc:1196] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/c63d20b1d7114e6fa591b5086e11690c
I20260925 12:17:13.696554 2010 tablet_bootstrap.cc:654] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:13.696575 2032 ts_tablet_manager.cc:1939] T c63d20b1d7114e6fa591b5086e11690c P 424446f817934078b5757ba12282c422: Deleting consensus metadata
I20260925 12:17:13.696898 1488 catalog_manager.cc:4841] TS 424446f817934078b5757ba12282c422 (127.0.98.3:41375): tablet c63d20b1d7114e6fa591b5086e11690c (table test-table [id=bc78918640c5445090ae7f0fe7f09b33]) successfully deleted
I20260925 12:17:13.697086 2030 ts_tablet_manager.cc:1939] T c63d20b1d7114e6fa591b5086e11690c P 2d925a59d87a4ba4a93b5ac7b459570e: Deleting consensus metadata
I20260925 12:17:13.697268 2010 tablet_bootstrap.cc:492] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422: No bootstrap required, opened a new log
I20260925 12:17:13.697320 2010 ts_tablet_manager.cc:1397] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:13.697475 2010 raft_consensus.cc:348] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } }
I20260925 12:17:13.697575 2010 raft_consensus.cc:374] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:13.697602 2010 raft_consensus.cc:729] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 424446f817934078b5757ba12282c422, State: Initialized, Role: FOLLOWER
I20260925 12:17:13.697639 2010 consensus_queue.cc:260] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } }
I20260925 12:17:13.697676 2031 ts_tablet_manager.cc:1905] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:13.697728 2010 ts_tablet_manager.cc:1428] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:13.697804 2007 tablet_bootstrap.cc:492] T aeb6290071ad4cd4a925e96ad543391f P 6c670ff22cbb498fbce3efc7a9499d8d: Bootstrap starting.
I20260925 12:17:13.697980 1491 catalog_manager.cc:4841] TS 2d925a59d87a4ba4a93b5ac7b459570e (127.0.98.1:45319): tablet c63d20b1d7114e6fa591b5086e11690c (table test-table [id=bc78918640c5445090ae7f0fe7f09b33]) successfully deleted
I20260925 12:17:13.698179 2007 tablet_bootstrap.cc:654] T aeb6290071ad4cd4a925e96ad543391f P 6c670ff22cbb498fbce3efc7a9499d8d: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:13.698520 2009 tablet_bootstrap.cc:492] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e: No bootstrap required, opened a new log
I20260925 12:17:13.698567 2009 ts_tablet_manager.cc:1397] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 12:17:13.698700 2009 raft_consensus.cc:348] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } }
I20260925 12:17:13.698781 2009 raft_consensus.cc:374] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:13.698804 2009 raft_consensus.cc:729] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d925a59d87a4ba4a93b5ac7b459570e, State: Initialized, Role: FOLLOWER
I20260925 12:17:13.698851 2009 consensus_queue.cc:260] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } }
I20260925 12:17:13.698939 2009 ts_tablet_manager.cc:1428] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:13.699107 2031 ts_tablet_manager.cc:1918] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:17:13.699167 2031 log.cc:1196] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/c63d20b1d7114e6fa591b5086e11690c
I20260925 12:17:13.699252 2007 tablet_bootstrap.cc:492] T aeb6290071ad4cd4a925e96ad543391f P 6c670ff22cbb498fbce3efc7a9499d8d: No bootstrap required, opened a new log
I20260925 12:17:13.699285 2007 ts_tablet_manager.cc:1397] T aeb6290071ad4cd4a925e96ad543391f P 6c670ff22cbb498fbce3efc7a9499d8d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:13.699388 2031 ts_tablet_manager.cc:1939] T c63d20b1d7114e6fa591b5086e11690c P 6c670ff22cbb498fbce3efc7a9499d8d: Deleting consensus metadata
I20260925 12:17:13.699409 2007 raft_consensus.cc:348] T aeb6290071ad4cd4a925e96ad543391f P 6c670ff22cbb498fbce3efc7a9499d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } }
I20260925 12:17:13.699471 2007 raft_consensus.cc:374] T aeb6290071ad4cd4a925e96ad543391f P 6c670ff22cbb498fbce3efc7a9499d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:13.699489 2007 raft_consensus.cc:729] T aeb6290071ad4cd4a925e96ad543391f P 6c670ff22cbb498fbce3efc7a9499d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c670ff22cbb498fbce3efc7a9499d8d, State: Initialized, Role: FOLLOWER
I20260925 12:17:13.699537 2007 consensus_queue.cc:260] T aeb6290071ad4cd4a925e96ad543391f P 6c670ff22cbb498fbce3efc7a9499d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } }
I20260925 12:17:13.699604 2007 ts_tablet_manager.cc:1428] T aeb6290071ad4cd4a925e96ad543391f P 6c670ff22cbb498fbce3efc7a9499d8d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:13.699714 1489 catalog_manager.cc:4841] TS 6c670ff22cbb498fbce3efc7a9499d8d (127.0.98.2:46003): tablet c63d20b1d7114e6fa591b5086e11690c (table test-table [id=bc78918640c5445090ae7f0fe7f09b33]) successfully deleted
W20260925 12:17:13.772401 1708 tablet.cc:2369] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:17:13.776517 2014 raft_consensus.cc:482] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:17:13.776609 2014 raft_consensus.cc:504] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } }
I20260925 12:17:13.776950 2014 leader_election.cc:288] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c670ff22cbb498fbce3efc7a9499d8d (127.0.98.2:46003), 424446f817934078b5757ba12282c422 (127.0.98.3:41375)
I20260925 12:17:13.780469 2037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.777106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46003 (local address 127.0.98.1:32789)
0925 12:17:13.777308 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:13.777318 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:13.777406 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:13.777568 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:13.777568 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:13.777570 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:13.777659 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.777661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.778590 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.778615 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.779683 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.779685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.780194 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.780195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.780265 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.780269 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:17:13.780345 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 12:17:13.780389 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":54,"threads_started":1}
I20260925 12:17:13.780542 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.777208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39221 (local address 127.0.98.3:41375)
0925 12:17:13.777251 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.777253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.777347 (+ 94us) server_negotiation.cc:407] Connection header received
0925 12:17:13.777427 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.777428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.777438 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.777482 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:13.777650 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.778246 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.779636 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.780205 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.780287 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.780296 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.780437 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.780475 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:17:13.780933 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.777219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32789 (local address 127.0.98.2:46003)
0925 12:17:13.777264 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.777266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.777380 (+ 114us) server_negotiation.cc:407] Connection header received
0925 12:17:13.777467 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.777468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.777477 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.777526 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:13.777739 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.778497 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.779756 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.780105 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.780181 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.780192 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.780795 (+ 603us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.780848 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 12:17:13.781072 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.777103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41375 (local address 127.0.98.1:39221)
0925 12:17:13.777269 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:13.777280 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:13.777363 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:13.777496 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:13.777497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:13.777499 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:13.777578 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.777580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.778326 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.778327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.779527 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.779529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.780287 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.780288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.780355 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.780358 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:13.780929 (+ 571us) client_negotiation.cc:240] Negotiation successful
0925 12:17:13.780981 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":56,"threads_started":1}
I20260925 12:17:13.781174 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb6290071ad4cd4a925e96ad543391f" candidate_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" is_pre_election: true
I20260925 12:17:13.781255 1801 raft_consensus.cc:2454] T aeb6290071ad4cd4a925e96ad543391f P 6c670ff22cbb498fbce3efc7a9499d8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d925a59d87a4ba4a93b5ac7b459570e in term 0.
I20260925 12:17:13.781378 1938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb6290071ad4cd4a925e96ad543391f" candidate_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "424446f817934078b5757ba12282c422" is_pre_election: true
I20260925 12:17:13.781415 1593 leader_election.cc:302] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d925a59d87a4ba4a93b5ac7b459570e, 6c670ff22cbb498fbce3efc7a9499d8d; no voters:
I20260925 12:17:13.781482 1938 raft_consensus.cc:2454] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d925a59d87a4ba4a93b5ac7b459570e in term 0.
I20260925 12:17:13.781546 2014 raft_consensus.cc:2781] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:17:13.781584 2014 raft_consensus.cc:482] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:17:13.781607 2014 raft_consensus.cc:3037] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:13.782274 2014 raft_consensus.cc:504] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } }
I20260925 12:17:13.782419 2014 leader_election.cc:288] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [CANDIDATE]: Term 1 election: Requested vote from peers 6c670ff22cbb498fbce3efc7a9499d8d (127.0.98.2:46003), 424446f817934078b5757ba12282c422 (127.0.98.3:41375)
I20260925 12:17:13.782531 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb6290071ad4cd4a925e96ad543391f" candidate_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c670ff22cbb498fbce3efc7a9499d8d"
I20260925 12:17:13.782603 1801 raft_consensus.cc:3037] T aeb6290071ad4cd4a925e96ad543391f P 6c670ff22cbb498fbce3efc7a9499d8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:13.782656 1938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb6290071ad4cd4a925e96ad543391f" candidate_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "424446f817934078b5757ba12282c422"
I20260925 12:17:13.782732 1938 raft_consensus.cc:3037] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:13.783339 1801 raft_consensus.cc:2454] T aeb6290071ad4cd4a925e96ad543391f P 6c670ff22cbb498fbce3efc7a9499d8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d925a59d87a4ba4a93b5ac7b459570e in term 1.
I20260925 12:17:13.783470 1938 raft_consensus.cc:2454] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d925a59d87a4ba4a93b5ac7b459570e in term 1.
I20260925 12:17:13.783663 1592 leader_election.cc:302] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d925a59d87a4ba4a93b5ac7b459570e, 424446f817934078b5757ba12282c422; no voters:
I20260925 12:17:13.783775 2014 raft_consensus.cc:2781] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:13.783874 2014 raft_consensus.cc:686] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [term 1 LEADER]: Becoming Leader. State: Replica: 2d925a59d87a4ba4a93b5ac7b459570e, State: Running, Role: LEADER
I20260925 12:17:13.783938 2014 consensus_queue.cc:237] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } }
I20260925 12:17:13.784588 1503 catalog_manager.cc:5495] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d925a59d87a4ba4a93b5ac7b459570e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } health_report { overall_health: UNKNOWN } } }
W20260925 12:17:13.805934 1988 tablet.cc:2369] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 12:17:13 dist-test-slave-clmx krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:13.818715 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.812232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36472 (local address 127.0.98.62:38751)
0925 12:17:13.812309 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.812312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.812407 (+ 95us) server_negotiation.cc:407] Connection header received
0925 12:17:13.812484 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.812485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.812494 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.812567 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:13.813036 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.813521 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.814209 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.814395 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.814465 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.816456 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:13.816457 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:13.816457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:13.816464 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:13.817585 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.817959 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.817960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.817960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.818014 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.818207 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.818208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.818208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.818314 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:13.818387 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.818532 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.818610 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 12:17:13.818780 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.812159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38751 (local address 127.0.0.1:36472)
0925 12:17:13.812330 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:13.812341 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:13.812425 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:13.812570 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:13.812571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:13.812906 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:13.812948 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.812950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.813598 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.813599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.814114 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.814116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.814473 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.814474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.814527 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.815053 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:13.815060 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:13.816367 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:13.817686 (+ 1319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:13.817693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:13.817695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:13.817874 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:13.818079 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:13.818080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:13.818080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:13.818146 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:13.818377 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:13.818383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:13.818425 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 12:17:13.818512 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 12:17:13.818663 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":62,"thread_start_us":102,"threads_started":1}
I20260925 12:17:13.820700 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36472:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 12:17:13.820894 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:17:13.822228 1641 tablet_service.cc:1467] Processing CreateTablet for tablet bd3ee86397194d759fc7488efc654fcb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=62d40707cadc4672a2754d872363fde3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 12:17:13.822285 1905 tablet_service.cc:1467] Processing CreateTablet for tablet bd3ee86397194d759fc7488efc654fcb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=62d40707cadc4672a2754d872363fde3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 12:17:13.822266 1781 tablet_service.cc:1467] Processing CreateTablet for tablet bd3ee86397194d759fc7488efc654fcb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=62d40707cadc4672a2754d872363fde3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 12:17:13.822366 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd3ee86397194d759fc7488efc654fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:13.822400 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd3ee86397194d759fc7488efc654fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:13.822400 1905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd3ee86397194d759fc7488efc654fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:13.823361 2009 tablet_bootstrap.cc:492] T bd3ee86397194d759fc7488efc654fcb P 2d925a59d87a4ba4a93b5ac7b459570e: Bootstrap starting.
I20260925 12:17:13.823789 2007 tablet_bootstrap.cc:492] T bd3ee86397194d759fc7488efc654fcb P 6c670ff22cbb498fbce3efc7a9499d8d: Bootstrap starting.
I20260925 12:17:13.823843 2009 tablet_bootstrap.cc:654] T bd3ee86397194d759fc7488efc654fcb P 2d925a59d87a4ba4a93b5ac7b459570e: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:13.824201 2007 tablet_bootstrap.cc:654] T bd3ee86397194d759fc7488efc654fcb P 6c670ff22cbb498fbce3efc7a9499d8d: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:13.824476 2009 tablet_bootstrap.cc:492] T bd3ee86397194d759fc7488efc654fcb P 2d925a59d87a4ba4a93b5ac7b459570e: No bootstrap required, opened a new log
I20260925 12:17:13.824551 2009 ts_tablet_manager.cc:1397] T bd3ee86397194d759fc7488efc654fcb P 2d925a59d87a4ba4a93b5ac7b459570e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:13.824565 2010 tablet_bootstrap.cc:492] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422: Bootstrap starting.
I20260925 12:17:13.824713 2009 raft_consensus.cc:348] T bd3ee86397194d759fc7488efc654fcb P 2d925a59d87a4ba4a93b5ac7b459570e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } }
I20260925 12:17:13.824810 2009 raft_consensus.cc:374] T bd3ee86397194d759fc7488efc654fcb P 2d925a59d87a4ba4a93b5ac7b459570e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:13.824842 2009 raft_consensus.cc:729] T bd3ee86397194d759fc7488efc654fcb P 2d925a59d87a4ba4a93b5ac7b459570e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d925a59d87a4ba4a93b5ac7b459570e, State: Initialized, Role: FOLLOWER
I20260925 12:17:13.824852 2007 tablet_bootstrap.cc:492] T bd3ee86397194d759fc7488efc654fcb P 6c670ff22cbb498fbce3efc7a9499d8d: No bootstrap required, opened a new log
I20260925 12:17:13.824852 2010 tablet_bootstrap.cc:654] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:13.824890 2007 ts_tablet_manager.cc:1397] T bd3ee86397194d759fc7488efc654fcb P 6c670ff22cbb498fbce3efc7a9499d8d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 12:17:13.824883 2009 consensus_queue.cc:260] T bd3ee86397194d759fc7488efc654fcb P 2d925a59d87a4ba4a93b5ac7b459570e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } }
I20260925 12:17:13.824980 2009 ts_tablet_manager.cc:1428] T bd3ee86397194d759fc7488efc654fcb P 2d925a59d87a4ba4a93b5ac7b459570e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:13.825052 2007 raft_consensus.cc:348] T bd3ee86397194d759fc7488efc654fcb P 6c670ff22cbb498fbce3efc7a9499d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } }
I20260925 12:17:13.825124 2007 raft_consensus.cc:374] T bd3ee86397194d759fc7488efc654fcb P 6c670ff22cbb498fbce3efc7a9499d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:13.825141 2007 raft_consensus.cc:729] T bd3ee86397194d759fc7488efc654fcb P 6c670ff22cbb498fbce3efc7a9499d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c670ff22cbb498fbce3efc7a9499d8d, State: Initialized, Role: FOLLOWER
I20260925 12:17:13.825103 2014 tablet_replica.cc:439] T bd3ee86397194d759fc7488efc654fcb P 2d925a59d87a4ba4a93b5ac7b459570e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } } }
I20260925 12:17:13.825194 2007 consensus_queue.cc:260] T bd3ee86397194d759fc7488efc654fcb P 6c670ff22cbb498fbce3efc7a9499d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } }
I20260925 12:17:13.825203 2014 tablet_replica.cc:442] T bd3ee86397194d759fc7488efc654fcb P 2d925a59d87a4ba4a93b5ac7b459570e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:17:13.825278 2007 ts_tablet_manager.cc:1428] T bd3ee86397194d759fc7488efc654fcb P 6c670ff22cbb498fbce3efc7a9499d8d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:13.825413 2013 tablet_replica.cc:439] T bd3ee86397194d759fc7488efc654fcb P 6c670ff22cbb498fbce3efc7a9499d8d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } } }
I20260925 12:17:13.825531 2013 tablet_replica.cc:442] T bd3ee86397194d759fc7488efc654fcb P 6c670ff22cbb498fbce3efc7a9499d8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:17:13.825955 2010 tablet_bootstrap.cc:492] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422: No bootstrap required, opened a new log
I20260925 12:17:13.826007 2010 ts_tablet_manager.cc:1397] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:13.826103 2010 raft_consensus.cc:348] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } }
I20260925 12:17:13.826185 2010 raft_consensus.cc:374] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:13.826207 2010 raft_consensus.cc:729] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 424446f817934078b5757ba12282c422, State: Initialized, Role: FOLLOWER
I20260925 12:17:13.826253 2010 consensus_queue.cc:260] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } }
I20260925 12:17:13.826342 2010 ts_tablet_manager.cc:1428] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:13.826364 2026 tablet_replica.cc:439] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } } }
I20260925 12:17:13.826457 2026 tablet_replica.cc:442] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:17:13.927834 2026 raft_consensus.cc:482] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:17:13.927917 2026 raft_consensus.cc:504] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } }
I20260925 12:17:13.928068 2026 leader_election.cc:288] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d925a59d87a4ba4a93b5ac7b459570e (127.0.98.1:45319), 6c670ff22cbb498fbce3efc7a9499d8d (127.0.98.2:46003)
I20260925 12:17:13.928231 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3ee86397194d759fc7488efc654fcb" candidate_uuid: "424446f817934078b5757ba12282c422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" is_pre_election: true
I20260925 12:17:13.928247 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3ee86397194d759fc7488efc654fcb" candidate_uuid: "424446f817934078b5757ba12282c422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" is_pre_election: true
I20260925 12:17:13.928337 1801 raft_consensus.cc:2454] T bd3ee86397194d759fc7488efc654fcb P 6c670ff22cbb498fbce3efc7a9499d8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 424446f817934078b5757ba12282c422 in term 0.
I20260925 12:17:13.928337 1661 raft_consensus.cc:2454] T bd3ee86397194d759fc7488efc654fcb P 2d925a59d87a4ba4a93b5ac7b459570e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 424446f817934078b5757ba12282c422 in term 0.
I20260925 12:17:13.928470 1875 leader_election.cc:302] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d925a59d87a4ba4a93b5ac7b459570e, 424446f817934078b5757ba12282c422; no voters:
I20260925 12:17:13.928574 2026 raft_consensus.cc:2781] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:17:13.928663 2026 raft_consensus.cc:482] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:17:13.928689 2026 raft_consensus.cc:3037] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:13.929374 2026 raft_consensus.cc:504] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } }
I20260925 12:17:13.929517 2026 leader_election.cc:288] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422 [CANDIDATE]: Term 1 election: Requested vote from peers 2d925a59d87a4ba4a93b5ac7b459570e (127.0.98.1:45319), 6c670ff22cbb498fbce3efc7a9499d8d (127.0.98.2:46003)
I20260925 12:17:13.929669 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3ee86397194d759fc7488efc654fcb" candidate_uuid: "424446f817934078b5757ba12282c422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d925a59d87a4ba4a93b5ac7b459570e"
I20260925 12:17:13.929672 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3ee86397194d759fc7488efc654fcb" candidate_uuid: "424446f817934078b5757ba12282c422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c670ff22cbb498fbce3efc7a9499d8d"
I20260925 12:17:13.929745 1801 raft_consensus.cc:3037] T bd3ee86397194d759fc7488efc654fcb P 6c670ff22cbb498fbce3efc7a9499d8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:13.929764 1661 raft_consensus.cc:3037] T bd3ee86397194d759fc7488efc654fcb P 2d925a59d87a4ba4a93b5ac7b459570e [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:13.930371 1801 raft_consensus.cc:2454] T bd3ee86397194d759fc7488efc654fcb P 6c670ff22cbb498fbce3efc7a9499d8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 424446f817934078b5757ba12282c422 in term 1.
I20260925 12:17:13.930428 1661 raft_consensus.cc:2454] T bd3ee86397194d759fc7488efc654fcb P 2d925a59d87a4ba4a93b5ac7b459570e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 424446f817934078b5757ba12282c422 in term 1.
I20260925 12:17:13.930562 1873 leader_election.cc:302] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 424446f817934078b5757ba12282c422, 6c670ff22cbb498fbce3efc7a9499d8d; no voters:
I20260925 12:17:13.930680 2026 raft_consensus.cc:2781] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:13.930725 2026 raft_consensus.cc:686] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422 [term 1 LEADER]: Becoming Leader. State: Replica: 424446f817934078b5757ba12282c422, State: Running, Role: LEADER
I20260925 12:17:13.930774 2026 consensus_queue.cc:237] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } }
I20260925 12:17:13.931057 2015 tablet_replica.cc:439] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422: TxnStatusTablet state changed. Reason: New leader 424446f817934078b5757ba12282c422. Latest consensus state: current_term: 1 leader_uuid: "424446f817934078b5757ba12282c422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } } }
I20260925 12:17:13.931159 2015 tablet_replica.cc:442] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422: This TxnStatusTablet replica's current role is: LEADER
I20260925 12:17:13.931334 2050 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 12:17:13.931349 1503 catalog_manager.cc:5495] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422 reported cstate change: term changed from 0 to 1, leader changed from <none> to 424446f817934078b5757ba12282c422 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "424446f817934078b5757ba12282c422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } health_report { overall_health: HEALTHY } } }
I20260925 12:17:13.931432 2050 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 12:17:13.941715 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.939316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52586 (local address 127.0.98.3:41375)
0925 12:17:13.939370 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.939372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.939452 (+ 80us) server_negotiation.cc:407] Connection header received
0925 12:17:13.939523 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.939524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.939537 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.939609 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:13.939743 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.940204 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.940974 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.941167 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.941250 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.941455 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:13.941490 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:13.941557 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.941621 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.941649 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 12:17:13.941735 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.939254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41375 (local address 127.0.0.1:52586)
0925 12:17:13.939381 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:13.939404 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:13.939464 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:13.939636 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:13.939637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:13.939638 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:17:13.939688 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.939691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.940284 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.940286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.940876 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.940877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.941253 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.941254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.941333 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.941367 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:17:13.941558 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:17:13.941567 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 12:17:13.941618 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 12:17:13.941666 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 12:17:13.942538 1801 raft_consensus.cc:1261] T bd3ee86397194d759fc7488efc654fcb P 6c670ff22cbb498fbce3efc7a9499d8d [term 1 FOLLOWER]: Refusing update from remote peer 424446f817934078b5757ba12282c422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:13.942562 1661 raft_consensus.cc:1261] T bd3ee86397194d759fc7488efc654fcb P 2d925a59d87a4ba4a93b5ac7b459570e [term 1 FOLLOWER]: Refusing update from remote peer 424446f817934078b5757ba12282c422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:13.942718 2015 consensus_queue.cc:1035] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:13.942786 2026 consensus_queue.cc:1035] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:13.943320 2013 tablet_replica.cc:439] T bd3ee86397194d759fc7488efc654fcb P 6c670ff22cbb498fbce3efc7a9499d8d: TxnStatusTablet state changed. Reason: New leader 424446f817934078b5757ba12282c422. Latest consensus state: current_term: 1 leader_uuid: "424446f817934078b5757ba12282c422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } } }
I20260925 12:17:13.943323 2014 tablet_replica.cc:439] T bd3ee86397194d759fc7488efc654fcb P 2d925a59d87a4ba4a93b5ac7b459570e: TxnStatusTablet state changed. Reason: New leader 424446f817934078b5757ba12282c422. Latest consensus state: current_term: 1 leader_uuid: "424446f817934078b5757ba12282c422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } } }
I20260925 12:17:13.943406 2013 tablet_replica.cc:442] T bd3ee86397194d759fc7488efc654fcb P 6c670ff22cbb498fbce3efc7a9499d8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:17:13.943413 2014 tablet_replica.cc:442] T bd3ee86397194d759fc7488efc654fcb P 2d925a59d87a4ba4a93b5ac7b459570e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:17:13.943782 2015 tablet_replica.cc:439] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "424446f817934078b5757ba12282c422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } } }
I20260925 12:17:13.943853 2015 tablet_replica.cc:442] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422: This TxnStatusTablet replica's current role is: LEADER
I20260925 12:17:13.943919 2020 mvcc.cc:205] Tried to move back new op lower bound from 7333227044627337216 to 7333227044580909056. Current Snapshot: MvccSnapshot[applied={T|T < 7333227044627337216 or (T in {7333227044627337216})}]
I20260925 12:17:13.944083 2017 tablet_replica.cc:439] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "424446f817934078b5757ba12282c422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } } }
I20260925 12:17:13.944175 2017 tablet_replica.cc:442] T bd3ee86397194d759fc7488efc654fcb P 424446f817934078b5757ba12282c422: This TxnStatusTablet replica's current role is: LEADER
I20260925 12:17:13.944617 2013 tablet_replica.cc:439] T bd3ee86397194d759fc7488efc654fcb P 6c670ff22cbb498fbce3efc7a9499d8d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "424446f817934078b5757ba12282c422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } } }
I20260925 12:17:13.944691 2013 tablet_replica.cc:442] T bd3ee86397194d759fc7488efc654fcb P 6c670ff22cbb498fbce3efc7a9499d8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:17:13.944962 2014 tablet_replica.cc:439] T bd3ee86397194d759fc7488efc654fcb P 2d925a59d87a4ba4a93b5ac7b459570e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "424446f817934078b5757ba12282c422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d925a59d87a4ba4a93b5ac7b459570e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45319 } } peers { permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 } } }
I20260925 12:17:13.945042 2014 tablet_replica.cc:442] T bd3ee86397194d759fc7488efc654fcb P 2d925a59d87a4ba4a93b5ac7b459570e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:17:13.948491 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.945710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38900 (local address 127.0.98.1:45319)
0925 12:17:13.945846 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.945848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.945857 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:17:13.945894 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.945895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.945910 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.945965 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:13.946146 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.946572 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.947658 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.947900 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.947971 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.948176 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:13.948269 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:13.948331 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.948395 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.948414 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 12:17:13.953110 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.950752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49789 (local address 127.0.98.3:41375)
0925 12:17:13.950786 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.950788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.950914 (+ 126us) server_negotiation.cc:407] Connection header received
0925 12:17:13.950974 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.950976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.950985 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.951039 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:13.951189 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.951657 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.952392 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.952587 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.952659 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.952849 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:13.952877 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:13.952955 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.953033 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.953050 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 12:17:13.953152 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.950695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41375 (local address 127.0.98.1:49789)
0925 12:17:13.950859 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:13.950869 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:13.950929 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:13.951043 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:13.951044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:13.951045 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:17:13.951110 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.951111 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.951718 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.951720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.952294 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.952296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.952682 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.952683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.952747 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.952759 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:17:13.952960 (+ 201us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:17:13.952961 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 12:17:13.953029 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 12:17:13.953067 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
I20260925 12:17:13.955071 1801 raft_consensus.cc:1261] T aeb6290071ad4cd4a925e96ad543391f P 6c670ff22cbb498fbce3efc7a9499d8d [term 1 FOLLOWER]: Refusing update from remote peer 2d925a59d87a4ba4a93b5ac7b459570e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:13.955144 1938 raft_consensus.cc:1261] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422 [term 1 FOLLOWER]: Refusing update from remote peer 2d925a59d87a4ba4a93b5ac7b459570e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:13.955309 2038 consensus_queue.cc:1035] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c670ff22cbb498fbce3efc7a9499d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:13.955391 2014 consensus_queue.cc:1035] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [LEADER]: Connected to new peer: Peer: permanent_uuid: "424446f817934078b5757ba12282c422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:13.963233 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.960488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45319 (local address 127.0.98.3:35859)
0925 12:17:13.960562 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:13.960572 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:13.960616 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:13.960835 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:13.960836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:13.960837 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:17:13.960865 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.960866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.961743 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.961744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.962454 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:13.962457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.962898 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.962899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:13.962944 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.962955 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:17:13.963122 (+ 167us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:17:13.963124 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:17:13.963156 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 12:17:13.963188 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 12:17:13.963233 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.960672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35859 (local address 127.0.98.1:45319)
0925 12:17:13.960727 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.960728 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.960737 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:17:13.960756 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.960757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.960766 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.960829 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:13.960903 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.961655 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.962528 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.962806 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.962875 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.963037 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:13.963066 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:13.963116 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.963158 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.963173 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 12:17:13.972332 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:13.972410 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:13.972999 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c985f96aa294557932e7b01e8687deb: Sending DeleteTablet for 3 replicas of tablet aeb6290071ad4cd4a925e96ad543391f
I20260925 12:17:13.973301 1905 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb6290071ad4cd4a925e96ad543391f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52568
I20260925 12:17:13.973316 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb6290071ad4cd4a925e96ad543391f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48080
I20260925 12:17:13.973392 2032 tablet_replica.cc:330] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422: stopping tablet replica
I20260925 12:17:13.973407 2031 tablet_replica.cc:330] T aeb6290071ad4cd4a925e96ad543391f P 6c670ff22cbb498fbce3efc7a9499d8d: stopping tablet replica
I20260925 12:17:13.973450 2031 raft_consensus.cc:2229] T aeb6290071ad4cd4a925e96ad543391f P 6c670ff22cbb498fbce3efc7a9499d8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:13.973451 2032 raft_consensus.cc:2229] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:13.973493 2032 raft_consensus.cc:2258] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:13.973492 2031 raft_consensus.cc:2258] T aeb6290071ad4cd4a925e96ad543391f P 6c670ff22cbb498fbce3efc7a9499d8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:13.973717 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb6290071ad4cd4a925e96ad543391f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38884
I20260925 12:17:13.973803 2030 tablet_replica.cc:330] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e: stopping tablet replica
I20260925 12:17:13.973816 2031 ts_tablet_manager.cc:1905] T aeb6290071ad4cd4a925e96ad543391f P 6c670ff22cbb498fbce3efc7a9499d8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:13.973840 2030 raft_consensus.cc:2229] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:17:13.973915 2030 raft_consensus.cc:2258] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:13.974200 2030 ts_tablet_manager.cc:1905] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:13.975453 2031 ts_tablet_manager.cc:1918] T aeb6290071ad4cd4a925e96ad543391f P 6c670ff22cbb498fbce3efc7a9499d8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:17:13.975548 2031 log.cc:1196] T aeb6290071ad4cd4a925e96ad543391f P 6c670ff22cbb498fbce3efc7a9499d8d: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/aeb6290071ad4cd4a925e96ad543391f
I20260925 12:17:13.975775 2030 ts_tablet_manager.cc:1918] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:17:13.975785 2032 ts_tablet_manager.cc:1905] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:13.975808 2031 ts_tablet_manager.cc:1939] T aeb6290071ad4cd4a925e96ad543391f P 6c670ff22cbb498fbce3efc7a9499d8d: Deleting consensus metadata
I20260925 12:17:13.975839 2030 log.cc:1196] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-0/wal/wals/aeb6290071ad4cd4a925e96ad543391f
I20260925 12:17:13.976042 2030 ts_tablet_manager.cc:1939] T aeb6290071ad4cd4a925e96ad543391f P 2d925a59d87a4ba4a93b5ac7b459570e: Deleting consensus metadata
I20260925 12:17:13.976130 1489 catalog_manager.cc:4841] TS 6c670ff22cbb498fbce3efc7a9499d8d (127.0.98.2:46003): tablet aeb6290071ad4cd4a925e96ad543391f (table test-table [id=712ade676549433cb9f3b2c49dcd81e3]) successfully deleted
I20260925 12:17:13.976822 1491 catalog_manager.cc:4841] TS 2d925a59d87a4ba4a93b5ac7b459570e (127.0.98.1:45319): tablet aeb6290071ad4cd4a925e96ad543391f (table test-table [id=712ade676549433cb9f3b2c49dcd81e3]) successfully deleted
I20260925 12:17:13.977300 2032 ts_tablet_manager.cc:1918] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:17:13.977353 2032 log.cc:1196] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/aeb6290071ad4cd4a925e96ad543391f
I20260925 12:17:13.977550 2032 ts_tablet_manager.cc:1939] T aeb6290071ad4cd4a925e96ad543391f P 424446f817934078b5757ba12282c422: Deleting consensus metadata
I20260925 12:17:13.977847 1488 catalog_manager.cc:4841] TS 424446f817934078b5757ba12282c422 (127.0.98.3:41375): tablet aeb6290071ad4cd4a925e96ad543391f (table test-table [id=712ade676549433cb9f3b2c49dcd81e3]) successfully deleted
Sep 25 12:17:13 dist-test-slave-clmx krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:17:13.986460 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.976920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38908 (local address 127.0.98.1:45319)
0925 12:17:13.976950 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.976951 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.977184 (+ 233us) server_negotiation.cc:407] Connection header received
0925 12:17:13.977227 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.977228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.977236 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.977314 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:13.977978 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.978626 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.979654 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.979926 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.979995 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.983232 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:13.983247 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:13.983247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:13.983255 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:13.985183 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.985580 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.985583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.985583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.985660 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.985882 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.985883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.985884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.986027 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:13.986174 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.986243 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.986350 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 12:17:13.986852 1701 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38908
I20260925 12:17:13.994930 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:13.987847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36478 (local address 127.0.98.62:38751)
0925 12:17:13.987884 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:13.987886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:13.987916 (+ 30us) server_negotiation.cc:407] Connection header received
0925 12:17:13.987985 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:13.987985 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:13.987994 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:13.988053 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:13.988636 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.989349 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.990288 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:13.990625 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:13.990691 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:13.992111 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:13.992112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:13.992113 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:13.992120 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:13.993812 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.994165 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.994166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.994166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.994222 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:13.994417 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:13.994418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:13.994418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:13.994546 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:13.994637 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:13.994768 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 12:17:13.994850 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 12:17:13.995250 1503 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36478
I20260925 12:17:13.995936 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 1576
I20260925 12:17:14.002823 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 1716
I20260925 12:17:14.007441 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 1856
I20260925 12:17:14.012508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 1472
2026-09-25T12:17:14Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2163 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:17:14 dist-test-slave-clmx krb5kdc[2072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:17:14 dist-test-slave-clmx krb5kdc[2072](info): set up 2 sockets
Sep 25 12:17:14 dist-test-slave-clmx krb5kdc[2072](info): commencing operation
krb5kdc: starting...
W20260925 12:17:15.007191 392 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:17:15 dist-test-slave-clmx krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:17:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:17:15.114440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36621
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:15.188386 2085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:15.188603 2085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:15.188639 2085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:15.189697 2085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:15.189759 2085 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 12:17:15.189781 2085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:15.189811 2085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:15.189832 2085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:15.189860 2085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:17:15.191180 2085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36621
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:15.191433 2085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:15.191702 2085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:15.196206 2091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:15.196261 2090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:15.196374 2085 server_base.cc:1034] running on GCE node
W20260925 12:17:15.196421 2093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:15.196661 2085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:15.196934 2085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:15.198071 2085 hybrid_clock.cc:648] HybridClock initialized: now 1790338635198061 us; error 45 us; skew 500 ppm
Sep 25 12:17:15 dist-test-slave-clmx krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:15.200266 2085 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:17:15.200649 2085 webserver.cc:460] Webserver started at http://127.0.98.62:38395/ using document root <none> and password file <none>
I20260925 12:17:15.200858 2085 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:15.200925 2085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:15.201097 2085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:15.202106 2085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "443dd8f6b0a341f6b9fbc72f794d6165"
format_stamp: "Formatted at 2026-09-25 12:17:15 on dist-test-slave-clmx"
server_key: "1fa68d11f1cddd2f5f377aaafd9f24a6"
server_key_iv: "8228ca7c61621c88b69013005e01aa2f"
server_key_version: "encryptionkey@0"
I20260925 12:17:15.202445 2085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "443dd8f6b0a341f6b9fbc72f794d6165"
format_stamp: "Formatted at 2026-09-25 12:17:15 on dist-test-slave-clmx"
server_key: "1fa68d11f1cddd2f5f377aaafd9f24a6"
server_key_iv: "8228ca7c61621c88b69013005e01aa2f"
server_key_version: "encryptionkey@0"
I20260925 12:17:15.203714 2085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:17:15.204392 2100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:15.204563 2085 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:15.204671 2085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "443dd8f6b0a341f6b9fbc72f794d6165"
format_stamp: "Formatted at 2026-09-25 12:17:15 on dist-test-slave-clmx"
server_key: "1fa68d11f1cddd2f5f377aaafd9f24a6"
server_key_iv: "8228ca7c61621c88b69013005e01aa2f"
server_key_version: "encryptionkey@0"
I20260925 12:17:15.204756 2085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:15.219484 2085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:15.221776 2085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:15.221935 2085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:15.226218 2085 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36621
I20260925 12:17:15.226305 2162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36621 every 8 connection(s)
I20260925 12:17:15.226593 2085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:15.227183 2163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:15.228379 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 2085
I20260925 12:17:15.228478 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:17:15.228636 392 external_mini_cluster.cc:1428] Setting key 358ca73bdbe7f705751d5080d7b50e8c
I20260925 12:17:15.229759 2163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165: Bootstrap starting.
I20260925 12:17:15.230345 2163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:15.230643 2163 log.cc:824] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:15.231355 2163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165: No bootstrap required, opened a new log
I20260925 12:17:15.233603 2163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443dd8f6b0a341f6b9fbc72f794d6165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36621 } }
I20260925 12:17:15.233741 2163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:15.233785 2163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443dd8f6b0a341f6b9fbc72f794d6165, State: Initialized, Role: FOLLOWER
I20260925 12:17:15.233919 2163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443dd8f6b0a341f6b9fbc72f794d6165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36621 } }
I20260925 12:17:15.233975 2163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:15.233999 2163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:15.234061 2163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:15.234494 2163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443dd8f6b0a341f6b9fbc72f794d6165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36621 } }
I20260925 12:17:15.234570 2163 leader_election.cc:302] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 443dd8f6b0a341f6b9fbc72f794d6165; no voters:
I20260925 12:17:15.234725 2163 leader_election.cc:288] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:17:15.234763 2168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:15.234867 2168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165 [term 1 LEADER]: Becoming Leader. State: Replica: 443dd8f6b0a341f6b9fbc72f794d6165, State: Running, Role: LEADER
I20260925 12:17:15.234953 2163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:15.234992 2168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443dd8f6b0a341f6b9fbc72f794d6165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36621 } }
Sep 25 12:17:15 dist-test-slave-clmx krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:15.235324 2168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 443dd8f6b0a341f6b9fbc72f794d6165. Latest consensus state: current_term: 1 leader_uuid: "443dd8f6b0a341f6b9fbc72f794d6165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443dd8f6b0a341f6b9fbc72f794d6165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36621 } } }
I20260925 12:17:15.235401 2168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:15.235551 2170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "443dd8f6b0a341f6b9fbc72f794d6165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443dd8f6b0a341f6b9fbc72f794d6165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36621 } } }
I20260925 12:17:15.235944 2170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165 [sys.catalog]: This master's current role is: LEADER
W20260925 12:17:15.236759 2183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:17:15.236821 2183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:17:15.236873 2175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:15.237013 2175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:17:15.238306 2175 catalog_manager.cc:1347] Generated new cluster ID: 216609882b904d7ab9770c9cae1636c0
I20260925 12:17:15.238351 2175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:17:15.238891 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.229139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35262 (local address 127.0.98.62:36621)
0925 12:17:15.229319 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:15.229327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:15.229360 (+ 33us) server_negotiation.cc:407] Connection header received
0925 12:17:15.229845 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:15.229852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:15.229904 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:15.230039 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:15.230582 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.231959 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.232694 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.233251 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.233357 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.235713 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:15.235716 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:15.235716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:15.235726 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:15.237399 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:15.237743 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:15.237746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:15.237746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:15.237840 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:15.238007 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:15.238008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:15.238008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:15.238270 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:15.238389 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:15.238617 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 12:17:15.238729 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260925 12:17:15.244704 2175 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:17:15.245355 2175 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:15.250465 2175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165: Generated new TSK 0
I20260925 12:17:15.250602 2175 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:17:15.272558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36621
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:15.351413 2189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:15.351599 2189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:15.351672 2189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:15.352828 2189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:15.352867 2189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:15.352888 2189 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 12:17:15.352900 2189 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 12:17:15.352917 2189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:17:15.354323 2189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:15.354558 2189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:15.354768 2189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:15.359489 2197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:15.359512 2195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:15.359524 2194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:15.359727 2189 server_base.cc:1034] running on GCE node
I20260925 12:17:15.359911 2189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:15.360133 2189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:15.361299 2189 hybrid_clock.cc:648] HybridClock initialized: now 1790338635361280 us; error 31 us; skew 500 ppm
Sep 25 12:17:15 dist-test-slave-clmx krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:15.363569 2189 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:17:15.363983 2189 webserver.cc:460] Webserver started at http://127.0.98.1:43277/ using document root <none> and password file <none>
I20260925 12:17:15.364224 2189 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:15.364322 2189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:15.364459 2189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:15.365432 2189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "9268de0b968a4b22a8a4306a276cdbc5"
format_stamp: "Formatted at 2026-09-25 12:17:15 on dist-test-slave-clmx"
server_key: "195bd912bd7747614780bba74dba9866"
server_key_iv: "d2ecaef73af778f83524bf9c44b2c69d"
server_key_version: "encryptionkey@0"
I20260925 12:17:15.365787 2189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9268de0b968a4b22a8a4306a276cdbc5"
format_stamp: "Formatted at 2026-09-25 12:17:15 on dist-test-slave-clmx"
server_key: "195bd912bd7747614780bba74dba9866"
server_key_iv: "d2ecaef73af778f83524bf9c44b2c69d"
server_key_version: "encryptionkey@0"
I20260925 12:17:15.367213 2189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:17:15.368040 2204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:15.368198 2189 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:15.368284 2189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "9268de0b968a4b22a8a4306a276cdbc5"
format_stamp: "Formatted at 2026-09-25 12:17:15 on dist-test-slave-clmx"
server_key: "195bd912bd7747614780bba74dba9866"
server_key_iv: "d2ecaef73af778f83524bf9c44b2c69d"
server_key_version: "encryptionkey@0"
I20260925 12:17:15.368360 2189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:15.381567 2189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:15.383145 2189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:15.383312 2189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:15.383921 2189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:15.383970 2189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:15.383999 2189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:15.384027 2189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 12:17:15 dist-test-slave-clmx krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:15.390959 2189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35461
I20260925 12:17:15.391038 2319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35461 every 8 connection(s)
I20260925 12:17:15.391340 2189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:17:15.393244 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.384187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34707 (local address 127.0.98.62:36621)
0925 12:17:15.384264 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:15.384266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:15.384443 (+ 177us) server_negotiation.cc:407] Connection header received
0925 12:17:15.384878 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:15.384880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:15.384895 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:15.384970 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:15.385581 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.386436 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.387973 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.388332 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.388416 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.390514 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:15.390516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:15.390516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:15.390524 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:15.392088 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:15.392442 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:15.392443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:15.392443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:15.392489 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:15.392711 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:15.392712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:15.392712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:15.392799 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:15.392844 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:15.393108 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 12:17:15.393167 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 12:17:15.393409 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.384168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36621 (local address 127.0.98.1:34707)
0925 12:17:15.384338 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:15.384368 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:15.384776 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:15.385033 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:15.385036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:15.385263 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:15.385438 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.385441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.386548 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.386549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.387866 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.387869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.388439 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.388440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.388603 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.389096 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:15.389109 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:15.390423 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:15.392159 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:15.392163 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:15.392164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:15.392381 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:15.392573 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:15.392574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:15.392574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:15.392661 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:15.392857 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:15.392861 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:15.392995 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 12:17:15.393086 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 12:17:15.393205 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":114,"threads_started":1}
I20260925 12:17:15.393913 2320 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36621
I20260925 12:17:15.393993 2320 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:15.394167 2320 heartbeater.cc:511] Master 127.0.98.62:36621 requested a full tablet report, sending...
I20260925 12:17:15.394701 2116 ts_manager.cc:194] Registered new tserver with Master: 9268de0b968a4b22a8a4306a276cdbc5 (127.0.98.1:35461)
I20260925 12:17:15.395728 2116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34707
I20260925 12:17:15.399799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 2189
I20260925 12:17:15.399897 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:17:15.399977 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.394870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42403 (local address 127.0.98.62:36621)
0925 12:17:15.394902 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:15.394904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:15.395016 (+ 112us) server_negotiation.cc:407] Connection header received
0925 12:17:15.395041 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:15.395042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:15.395051 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:15.395111 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:15.395373 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.396037 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.396807 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.397010 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.397102 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.397855 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:15.397856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:15.397856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:15.397863 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:15.399059 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:15.399333 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:15.399333 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:15.399334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:15.399372 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:15.399521 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:15.399521 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:15.399521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:15.399602 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:15.399716 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:15.399820 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 12:17:15.399893 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 12:17:15.399998 2326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.394834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36621 (local address 127.0.98.1:42403)
0925 12:17:15.394947 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:15.394955 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:15.395011 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:15.395112 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:15.395113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:15.395236 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:15.395282 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.395283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.396119 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.396120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.396698 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.396701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.397103 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.397104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.397178 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.397507 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:15.397511 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:15.397756 (+ 245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:15.399127 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:15.399128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:15.399128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:15.399270 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:15.399429 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:15.399429 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:15.399430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:15.399479 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:15.399706 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:15.399707 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:15.399737 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 12:17:15.399785 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 12:17:15.399850 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":26,"thread_start_us":72,"threads_started":1}
I20260925 12:17:15.400080 392 external_mini_cluster.cc:1428] Setting key 3371f338975d6d4b6daa918d6790b24c
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:17:15.422170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36621
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:15.501573 2329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:15.501784 2329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:15.501821 2329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:15.502964 2329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:15.503034 2329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:15.503080 2329 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 12:17:15.503129 2329 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 12:17:15.503162 2329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:17:15.504619 2329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:15.504868 2329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:15.505095 2329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:15.509857 2334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:15.509851 2335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:15.509859 2337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:15.510080 2329 server_base.cc:1034] running on GCE node
I20260925 12:17:15.510245 2329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:15.510484 2329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:15.511660 2329 hybrid_clock.cc:648] HybridClock initialized: now 1790338635511637 us; error 33 us; skew 500 ppm
Sep 25 12:17:15 dist-test-slave-clmx krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:15.513880 2329 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:17:15.514250 2329 webserver.cc:460] Webserver started at http://127.0.98.2:41911/ using document root <none> and password file <none>
I20260925 12:17:15.514467 2329 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:15.514531 2329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:15.514663 2329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:15.515579 2329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "04884f65bae6405f806102a42e6344a9"
format_stamp: "Formatted at 2026-09-25 12:17:15 on dist-test-slave-clmx"
server_key: "47c374b011ab4e647ff94a3435d61d17"
server_key_iv: "ebfc07a4bf6030bcdbf0232d04643620"
server_key_version: "encryptionkey@0"
I20260925 12:17:15.516004 2329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "04884f65bae6405f806102a42e6344a9"
format_stamp: "Formatted at 2026-09-25 12:17:15 on dist-test-slave-clmx"
server_key: "47c374b011ab4e647ff94a3435d61d17"
server_key_iv: "ebfc07a4bf6030bcdbf0232d04643620"
server_key_version: "encryptionkey@0"
I20260925 12:17:15.517232 2329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:15.517875 2344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:15.518054 2329 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:15.518132 2329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "04884f65bae6405f806102a42e6344a9"
format_stamp: "Formatted at 2026-09-25 12:17:15 on dist-test-slave-clmx"
server_key: "47c374b011ab4e647ff94a3435d61d17"
server_key_iv: "ebfc07a4bf6030bcdbf0232d04643620"
server_key_version: "encryptionkey@0"
I20260925 12:17:15.518203 2329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:15.539284 2329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:15.540861 2329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:15.541023 2329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:15.541457 2329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:15.541522 2329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:15.541564 2329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:15.541594 2329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 12:17:15 dist-test-slave-clmx krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:15.548398 2329 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36637
I20260925 12:17:15.548477 2459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36637 every 8 connection(s)
I20260925 12:17:15.548746 2329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:17:15.550668 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.541953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36169 (local address 127.0.98.62:36621)
0925 12:17:15.542017 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:15.542019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:15.542229 (+ 210us) server_negotiation.cc:407] Connection header received
0925 12:17:15.542645 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:15.542646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:15.542658 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:15.542731 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:15.543526 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.544400 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.545543 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.545819 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.545883 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.548524 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:15.548525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:15.548526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:15.548533 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:15.549766 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:15.550084 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:15.550085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:15.550085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:15.550128 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:15.550268 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:15.550269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:15.550269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:15.550348 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:15.550388 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:15.550561 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 12:17:15.550614 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 12:17:15.550828 2353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.541935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36621 (local address 127.0.98.2:36169)
0925 12:17:15.542099 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:15.542143 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:15.542549 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:15.542790 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:15.542795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:15.543134 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:15.543408 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.543413 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.544509 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.544510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.545423 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.545425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.545920 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.545921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.546091 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.546609 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:15.546617 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:15.548447 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:15.549833 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:15.549837 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:15.549839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:15.550021 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:15.550165 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:15.550165 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:15.550165 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:15.550236 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:15.550379 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:15.550381 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:15.550466 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 12:17:15.550530 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 12:17:15.550626 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260925 12:17:15.551326 2460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36621
I20260925 12:17:15.551404 2460 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:15.551581 2460 heartbeater.cc:511] Master 127.0.98.62:36621 requested a full tablet report, sending...
I20260925 12:17:15.551949 2116 ts_manager.cc:194] Registered new tserver with Master: 04884f65bae6405f806102a42e6344a9 (127.0.98.2:36637)
I20260925 12:17:15.552399 2116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36169
I20260925 12:17:15.556326 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 2329
I20260925 12:17:15.556432 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:17:15.556620 392 external_mini_cluster.cc:1428] Setting key 6de95e9a3b81644e55d3601e1ffc373d
I20260925 12:17:15.557415 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.552276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36621 (local address 127.0.98.2:55195)
0925 12:17:15.552455 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:15.552466 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:15.552541 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:15.552709 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:15.552710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:15.552855 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:15.552901 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.552902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.553544 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.553545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.554034 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.554035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.554323 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.554324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.554376 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.554685 (+ 309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:15.554688 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:15.554921 (+ 233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:15.556146 (+ 1225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:15.556147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:15.556147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:15.556333 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:15.556550 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:15.556550 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:15.556551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:15.556626 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:15.557108 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:15.557108 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:15.557164 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 12:17:15.557234 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 12:17:15.557350 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":95,"threads_started":1}
I20260925 12:17:15.557500 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.552361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55195 (local address 127.0.98.62:36621)
0925 12:17:15.552394 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:15.552396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:15.552518 (+ 122us) server_negotiation.cc:407] Connection header received
0925 12:17:15.552620 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:15.552621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:15.552630 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:15.552707 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:15.552971 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.553462 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.554100 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.554262 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.554308 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.554991 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:15.554992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:15.554992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:15.554998 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:15.556070 (+ 1072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:15.556414 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:15.556415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:15.556415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:15.556481 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:15.556693 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:15.556694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:15.556694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:15.556871 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:15.556950 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:15.557293 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 12:17:15.557388 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:17:15.579402 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36621
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:15.655869 2469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:15.656090 2469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:15.656121 2469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:15.657698 2469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:15.657750 2469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:15.657780 2469 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 12:17:15.657797 2469 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 12:17:15.657823 2469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:17:15.659363 2469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:15.659523 2469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:15.659755 2469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:15.664266 2475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:15.664318 2477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:15.664274 2474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:15.664461 2469 server_base.cc:1034] running on GCE node
I20260925 12:17:15.664652 2469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:15.664891 2469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:15.666050 2469 hybrid_clock.cc:648] HybridClock initialized: now 1790338635666031 us; error 31 us; skew 500 ppm
Sep 25 12:17:15 dist-test-slave-clmx krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:15.668414 2469 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:17:15.668866 2469 webserver.cc:460] Webserver started at http://127.0.98.3:37135/ using document root <none> and password file <none>
I20260925 12:17:15.669080 2469 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:15.669143 2469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:15.669301 2469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:15.670218 2469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "152e2c85bff341db9e4f6b7347af8034"
format_stamp: "Formatted at 2026-09-25 12:17:15 on dist-test-slave-clmx"
server_key: "13f4ec8df2c4955d557173b469abddfb"
server_key_iv: "357ae0c8ecef119520245920ab16357f"
server_key_version: "encryptionkey@0"
I20260925 12:17:15.670544 2469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "152e2c85bff341db9e4f6b7347af8034"
format_stamp: "Formatted at 2026-09-25 12:17:15 on dist-test-slave-clmx"
server_key: "13f4ec8df2c4955d557173b469abddfb"
server_key_iv: "357ae0c8ecef119520245920ab16357f"
server_key_version: "encryptionkey@0"
I20260925 12:17:15.671939 2469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:17:15.672603 2484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:15.672762 2469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:15.672851 2469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "152e2c85bff341db9e4f6b7347af8034"
format_stamp: "Formatted at 2026-09-25 12:17:15 on dist-test-slave-clmx"
server_key: "13f4ec8df2c4955d557173b469abddfb"
server_key_iv: "357ae0c8ecef119520245920ab16357f"
server_key_version: "encryptionkey@0"
I20260925 12:17:15.672935 2469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:15.685475 2469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:15.687067 2469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:15.687224 2469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:15.687760 2469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:15.687819 2469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:15.687851 2469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:15.687879 2469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 12:17:15 dist-test-slave-clmx krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:15.694662 2469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45487
I20260925 12:17:15.694706 2599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45487 every 8 connection(s)
I20260925 12:17:15.694988 2469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:17:15.696939 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.688119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42725 (local address 127.0.98.62:36621)
0925 12:17:15.688180 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:15.688182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:15.688452 (+ 270us) server_negotiation.cc:407] Connection header received
0925 12:17:15.689018 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:15.689019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:15.689028 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:15.689074 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:15.689817 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.690364 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.691372 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.691586 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.691701 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.694460 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:15.694462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:15.694462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:15.694469 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:15.695965 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:15.696288 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:15.696288 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:15.696289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:15.696332 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:15.696515 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:15.696516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:15.696516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:15.696607 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:15.696651 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:15.696836 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 12:17:15.696885 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 12:17:15.697077 2493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.688108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36621 (local address 127.0.98.3:42725)
0925 12:17:15.688315 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:15.688359 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:15.688923 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:15.689140 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:15.689144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:15.689476 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:15.689735 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.689741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.690463 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.690464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.691264 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.691265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.691711 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.691712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.691827 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.692338 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:15.692350 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:15.694359 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:15.696034 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:15.696038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:15.696039 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:15.696224 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:15.696385 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:15.696386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:15.696386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:15.696466 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:15.696663 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:15.696666 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:15.696756 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 12:17:15.696825 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 12:17:15.696921 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":109,"threads_started":1}
I20260925 12:17:15.697547 2600 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36621
I20260925 12:17:15.697631 2600 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:15.697774 2600 heartbeater.cc:511] Master 127.0.98.62:36621 requested a full tablet report, sending...
I20260925 12:17:15.698099 2117 ts_manager.cc:194] Registered new tserver with Master: 152e2c85bff341db9e4f6b7347af8034 (127.0.98.3:45487)
I20260925 12:17:15.698526 2117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42725
I20260925 12:17:15.703210 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.698479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54109 (local address 127.0.98.62:36621)
0925 12:17:15.698517 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:15.698518 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:15.698615 (+ 97us) server_negotiation.cc:407] Connection header received
0925 12:17:15.698633 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:15.698633 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:15.698641 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:15.698682 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:15.698987 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.699395 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.700158 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.700355 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.700431 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.701191 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:15.701193 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:15.701193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:15.701200 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:15.702252 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:15.702551 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:15.702552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:15.702552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:15.702606 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:15.702776 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:15.702777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:15.702778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:15.702870 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:15.702951 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:15.703075 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 12:17:15.703136 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:17:15.703248 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.698423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36621 (local address 127.0.98.3:54109)
0925 12:17:15.698523 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:15.698531 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:15.698585 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:15.698682 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:15.698683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:15.698858 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:15.698904 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.698906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.699481 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.699482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.700043 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.700044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.700432 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.700433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.700497 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.700825 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:15.700829 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:15.701091 (+ 262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:15.702337 (+ 1246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:15.702338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:15.702339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:15.702480 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:15.702663 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:15.702663 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:15.702664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:15.702711 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:15.702945 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:15.702946 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:15.702981 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 12:17:15.703042 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 12:17:15.703157 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":62,"threads_started":1}
I20260925 12:17:15.703279 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 2469
I20260925 12:17:15.703385 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:17:15.703552 392 external_mini_cluster.cc:1428] Setting key 39dec6a7d8eebf777f5b599e4381f7d1
I20260925 12:17:15.704774 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 12:17:15 dist-test-slave-clmx krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 12:17:15 dist-test-slave-clmx krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:15.724929 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.719086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.0.98.62:36621)
0925 12:17:15.719138 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:15.719144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:15.719151 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:17:15.719165 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:15.719165 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:15.719171 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:15.719212 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:15.719712 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.720204 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.720840 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.720992 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.721096 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.722999 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:15.723000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:15.723001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:15.723005 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:15.724068 (+ 1063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:15.724378 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:15.724379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:15.724379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:15.724423 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:15.724598 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:15.724598 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:15.724598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:15.724682 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:15.724751 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:15.724832 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 12:17:15.724879 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 12:17:15.726296 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:17:15.727710 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:17:15.733453 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.729686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35461 (local address 127.0.0.1:56214)
0925 12:17:15.729816 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:15.729848 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:15.729903 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:15.730207 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:15.730210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:15.730212 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:15.730271 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.730272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.731309 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.731310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.732282 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.732284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.733200 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.733201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.733273 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.733284 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 12:17:15.733352 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 12:17:15.733392 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":42,"threads_started":1}
I20260925 12:17:15.733690 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.729772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56214 (local address 127.0.98.1:35461)
0925 12:17:15.730032 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:15.730037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:15.730048 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:17:15.730100 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:15.730103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:15.730118 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:15.730191 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:15.730326 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.731227 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.732413 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.733100 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.733171 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.733230 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:15.733380 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 12:17:15.733415 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":80,"threads_started":1}
I20260925 12:17:15.733867 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.729686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36637 (local address 127.0.0.1:37906)
0925 12:17:15.730079 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:15.730088 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:15.730148 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:15.730383 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:15.730402 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:15.730404 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:15.730447 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.730448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.731320 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.731322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.732665 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.732667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.733654 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.733655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.733714 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.733718 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:17:15.733767 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 12:17:15.733803 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":41,"thread_start_us":56,"threads_started":1}
I20260925 12:17:15.734107 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.729886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37906 (local address 127.0.98.2:36637)
0925 12:17:15.730039 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:15.730043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:15.730159 (+ 116us) server_negotiation.cc:407] Connection header received
0925 12:17:15.730215 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:15.730217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:15.730232 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:15.730309 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:15.730499 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.731231 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.732793 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.733513 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.733579 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.733626 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:15.733963 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 12:17:15.734008 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260925 12:17:15.734395 2254 tablet_service.cc:1467] Processing CreateTablet for tablet 03e32a3e08fd498481b4419f03a38559 (DEFAULT_TABLE table=test-table [id=b260325130fa45f49f2a8bfbd6806f5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:15.734741 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.729704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45487 (local address 127.0.0.1:44326)
0925 12:17:15.730225 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:15.730233 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:15.730273 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:15.731459 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:15.731460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:15.731462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:15.731598 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.731600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.732807 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.732808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.733682 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.733684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.734537 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.734538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.734590 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.734593 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:15.734641 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 12:17:15.734674 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":93,"thread_start_us":57,"threads_started":1}
I20260925 12:17:15.734742 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e32a3e08fd498481b4419f03a38559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:15.734762 2394 tablet_service.cc:1467] Processing CreateTablet for tablet 03e32a3e08fd498481b4419f03a38559 (DEFAULT_TABLE table=test-table [id=b260325130fa45f49f2a8bfbd6806f5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:15.734995 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.729856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44326 (local address 127.0.98.3:45487)
0925 12:17:15.729987 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:15.729990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:15.730274 (+ 284us) server_negotiation.cc:407] Connection header received
0925 12:17:15.731355 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:15.731357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:15.731371 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:15.731444 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:15.731723 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.732733 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.733758 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.734396 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.734458 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.734504 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:15.734670 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 12:17:15.734706 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260925 12:17:15.735070 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e32a3e08fd498481b4419f03a38559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:15.735692 2534 tablet_service.cc:1467] Processing CreateTablet for tablet 03e32a3e08fd498481b4419f03a38559 (DEFAULT_TABLE table=test-table [id=b260325130fa45f49f2a8bfbd6806f5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:15.735947 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e32a3e08fd498481b4419f03a38559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:15.737694 2620 tablet_bootstrap.cc:492] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5: Bootstrap starting.
I20260925 12:17:15.738334 2620 tablet_bootstrap.cc:654] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:15.738442 2621 tablet_bootstrap.cc:492] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9: Bootstrap starting.
I20260925 12:17:15.738626 2620 log.cc:824] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:15.739015 2621 tablet_bootstrap.cc:654] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:15.739158 2623 tablet_bootstrap.cc:492] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034: Bootstrap starting.
I20260925 12:17:15.739307 2620 tablet_bootstrap.cc:492] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5: No bootstrap required, opened a new log
I20260925 12:17:15.739306 2621 log.cc:824] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:15.739372 2620 ts_tablet_manager.cc:1397] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:17:15.739940 2623 tablet_bootstrap.cc:654] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:15.740023 2621 tablet_bootstrap.cc:492] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9: No bootstrap required, opened a new log
I20260925 12:17:15.740084 2621 ts_tablet_manager.cc:1397] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:17:15.740250 2623 log.cc:824] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:15.740810 2623 tablet_bootstrap.cc:492] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034: No bootstrap required, opened a new log
I20260925 12:17:15.740909 2623 ts_tablet_manager.cc:1397] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 12:17:15.741221 2620 raft_consensus.cc:348] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:15.741379 2620 raft_consensus.cc:374] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:15.741412 2620 raft_consensus.cc:729] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9268de0b968a4b22a8a4306a276cdbc5, State: Initialized, Role: FOLLOWER
I20260925 12:17:15.741513 2620 consensus_queue.cc:260] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:15.741755 2620 ts_tablet_manager.cc:1428] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 12:17:15.741783 2320 heartbeater.cc:503] Master 127.0.98.62:36621 was elected leader, sending a full tablet report...
I20260925 12:17:15.741921 2621 raft_consensus.cc:348] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:15.742029 2621 raft_consensus.cc:374] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:15.742048 2621 raft_consensus.cc:729] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04884f65bae6405f806102a42e6344a9, State: Initialized, Role: FOLLOWER
I20260925 12:17:15.742166 2621 consensus_queue.cc:260] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:15.742412 2621 ts_tablet_manager.cc:1428] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 12:17:15.742422 2460 heartbeater.cc:503] Master 127.0.98.62:36621 was elected leader, sending a full tablet report...
I20260925 12:17:15.742908 2623 raft_consensus.cc:348] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:15.743049 2623 raft_consensus.cc:374] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:15.743094 2623 raft_consensus.cc:729] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152e2c85bff341db9e4f6b7347af8034, State: Initialized, Role: FOLLOWER
I20260925 12:17:15.743203 2623 consensus_queue.cc:260] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:15.743455 2623 ts_tablet_manager.cc:1428] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 12:17:15.743520 2600 heartbeater.cc:503] Master 127.0.98.62:36621 was elected leader, sending a full tablet report...
I20260925 12:17:15.785441 2626 raft_consensus.cc:482] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:17:15.785537 2626 raft_consensus.cc:504] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:15.785836 2626 leader_election.cc:288] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04884f65bae6405f806102a42e6344a9 (127.0.98.2:36637), 152e2c85bff341db9e4f6b7347af8034 (127.0.98.3:45487)
I20260925 12:17:15.789505 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.786101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40491 (local address 127.0.98.3:45487)
0925 12:17:15.786149 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:15.786151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:15.786262 (+ 111us) server_negotiation.cc:407] Connection header received
0925 12:17:15.786299 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:15.786300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:15.786309 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:15.786360 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:15.786496 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.787231 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.788641 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.789163 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.789232 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.789241 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:15.789409 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 12:17:15.789443 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 12:17:15.789505 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.786084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35695 (local address 127.0.98.2:36637)
0925 12:17:15.786126 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:15.786128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:15.786139 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:17:15.786220 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:15.786220 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:15.786229 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:15.786304 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:15.786503 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.787224 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.788640 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.789159 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.789232 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.789241 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:15.789409 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 12:17:15.789443 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:17:15.789541 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.786019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36637 (local address 127.0.98.1:35695)
0925 12:17:15.786078 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:15.786089 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:15.786144 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:15.786329 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:15.786330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:15.786352 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:15.786418 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.786420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.787305 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.787306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.788555 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.788557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.789248 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.789249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.789329 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.789331 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:17:15.789384 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 12:17:15.789432 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":16}
I20260925 12:17:15.789549 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.786019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45487 (local address 127.0.98.1:40491)
0925 12:17:15.786192 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:15.786201 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:15.786255 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:15.786372 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:15.786372 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:15.786374 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:15.786424 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.786425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.787305 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.787306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.788554 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.788557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.789249 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.789249 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.789325 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.789329 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:17:15.789384 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 12:17:15.789432 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":120,"thread_start_us":79,"threads_started":1}
I20260925 12:17:15.789877 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e32a3e08fd498481b4419f03a38559" candidate_uuid: "9268de0b968a4b22a8a4306a276cdbc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04884f65bae6405f806102a42e6344a9" is_pre_election: true
I20260925 12:17:15.789886 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e32a3e08fd498481b4419f03a38559" candidate_uuid: "9268de0b968a4b22a8a4306a276cdbc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152e2c85bff341db9e4f6b7347af8034" is_pre_election: true
I20260925 12:17:15.790000 2414 raft_consensus.cc:2454] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9268de0b968a4b22a8a4306a276cdbc5 in term 0.
I20260925 12:17:15.790035 2553 raft_consensus.cc:2454] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9268de0b968a4b22a8a4306a276cdbc5 in term 0.
I20260925 12:17:15.790220 2206 leader_election.cc:302] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04884f65bae6405f806102a42e6344a9, 9268de0b968a4b22a8a4306a276cdbc5; no voters:
I20260925 12:17:15.790337 2626 raft_consensus.cc:2781] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:17:15.790390 2626 raft_consensus.cc:482] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:17:15.790405 2626 raft_consensus.cc:3037] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:15.791020 2626 raft_consensus.cc:504] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:15.791153 2626 leader_election.cc:288] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [CANDIDATE]: Term 1 election: Requested vote from peers 04884f65bae6405f806102a42e6344a9 (127.0.98.2:36637), 152e2c85bff341db9e4f6b7347af8034 (127.0.98.3:45487)
I20260925 12:17:15.791272 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e32a3e08fd498481b4419f03a38559" candidate_uuid: "9268de0b968a4b22a8a4306a276cdbc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04884f65bae6405f806102a42e6344a9"
I20260925 12:17:15.791309 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e32a3e08fd498481b4419f03a38559" candidate_uuid: "9268de0b968a4b22a8a4306a276cdbc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152e2c85bff341db9e4f6b7347af8034"
I20260925 12:17:15.791350 2414 raft_consensus.cc:3037] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:15.791369 2553 raft_consensus.cc:3037] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:15.792182 2553 raft_consensus.cc:2454] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9268de0b968a4b22a8a4306a276cdbc5 in term 1.
I20260925 12:17:15.792182 2414 raft_consensus.cc:2454] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9268de0b968a4b22a8a4306a276cdbc5 in term 1.
I20260925 12:17:15.792618 2205 leader_election.cc:302] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 152e2c85bff341db9e4f6b7347af8034, 9268de0b968a4b22a8a4306a276cdbc5; no voters:
I20260925 12:17:15.792761 2626 raft_consensus.cc:2781] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:15.792876 2626 raft_consensus.cc:686] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [term 1 LEADER]: Becoming Leader. State: Replica: 9268de0b968a4b22a8a4306a276cdbc5, State: Running, Role: LEADER
I20260925 12:17:15.792961 2626 consensus_queue.cc:237] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:15.793753 2116 catalog_manager.cc:5495] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9268de0b968a4b22a8a4306a276cdbc5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9268de0b968a4b22a8a4306a276cdbc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } health_report { overall_health: UNKNOWN } } }
W20260925 12:17:15.799327 2461 tablet.cc:2369] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 12:17:15 dist-test-slave-clmx krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:15.829893 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.824112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36621 (local address 127.0.0.1:35284)
0925 12:17:15.824295 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:15.824304 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:15.824356 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:15.824469 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:15.824470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:15.824733 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:15.824769 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.824771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.825330 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.825330 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.825830 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:15.825831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.826114 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.826115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:15.826149 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.826660 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:15.826664 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:15.827913 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:15.829079 (+ 1166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:15.829084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:15.829086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:15.829247 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:15.829408 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:15.829409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:15.829409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:15.829455 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:15.829633 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:15.829637 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:15.829688 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 12:17:15.829743 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 12:17:15.829818 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":112,"threads_started":1}
I20260925 12:17:15.829962 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:15.824190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.0.98.62:36621)
0925 12:17:15.824253 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:15.824255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:15.824357 (+ 102us) server_negotiation.cc:407] Connection header received
0925 12:17:15.824392 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:15.824393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:15.824400 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:15.824450 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:15.824830 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.825267 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.825891 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:15.826049 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:15.826093 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:15.827980 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:15.827981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:15.827981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:15.827986 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:15.829015 (+ 1029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:15.829307 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:15.829307 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:15.829308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:15.829359 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:15.829500 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:15.829500 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:15.829501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:15.829579 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:15.829627 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:15.829771 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 12:17:15.829856 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":46}
I20260925 12:17:15.831477 2116 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35284:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 12:17:15.831691 2116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:17:15.833033 2254 tablet_service.cc:1467] Processing CreateTablet for tablet d39290228be9427e9c197eb68b4347b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=57b968c805bb4bd599cda0c9864a1d44]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 12:17:15.833045 2534 tablet_service.cc:1467] Processing CreateTablet for tablet d39290228be9427e9c197eb68b4347b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=57b968c805bb4bd599cda0c9864a1d44]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 12:17:15.833204 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d39290228be9427e9c197eb68b4347b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:15.833212 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d39290228be9427e9c197eb68b4347b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:15.834151 2623 tablet_bootstrap.cc:492] T d39290228be9427e9c197eb68b4347b0 P 152e2c85bff341db9e4f6b7347af8034: Bootstrap starting.
I20260925 12:17:15.834170 2620 tablet_bootstrap.cc:492] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5: Bootstrap starting.
I20260925 12:17:15.834240 2394 tablet_service.cc:1467] Processing CreateTablet for tablet d39290228be9427e9c197eb68b4347b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=57b968c805bb4bd599cda0c9864a1d44]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 12:17:15.834357 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d39290228be9427e9c197eb68b4347b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:15.834513 2620 tablet_bootstrap.cc:654] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:15.834645 2623 tablet_bootstrap.cc:654] T d39290228be9427e9c197eb68b4347b0 P 152e2c85bff341db9e4f6b7347af8034: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:15.835145 2620 tablet_bootstrap.cc:492] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5: No bootstrap required, opened a new log
I20260925 12:17:15.835196 2620 ts_tablet_manager.cc:1397] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 12:17:15.835233 2623 tablet_bootstrap.cc:492] T d39290228be9427e9c197eb68b4347b0 P 152e2c85bff341db9e4f6b7347af8034: No bootstrap required, opened a new log
I20260925 12:17:15.835276 2623 ts_tablet_manager.cc:1397] T d39290228be9427e9c197eb68b4347b0 P 152e2c85bff341db9e4f6b7347af8034: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 12:17:15.835328 2620 raft_consensus.cc:348] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:15.835422 2620 raft_consensus.cc:374] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:15.835450 2620 raft_consensus.cc:729] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9268de0b968a4b22a8a4306a276cdbc5, State: Initialized, Role: FOLLOWER
I20260925 12:17:15.835418 2623 raft_consensus.cc:348] T d39290228be9427e9c197eb68b4347b0 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:15.835486 2623 raft_consensus.cc:374] T d39290228be9427e9c197eb68b4347b0 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:15.835510 2623 raft_consensus.cc:729] T d39290228be9427e9c197eb68b4347b0 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152e2c85bff341db9e4f6b7347af8034, State: Initialized, Role: FOLLOWER
I20260925 12:17:15.835489 2620 consensus_queue.cc:260] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:15.835574 2620 ts_tablet_manager.cc:1428] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 12:17:15.835552 2623 consensus_queue.cc:260] T d39290228be9427e9c197eb68b4347b0 P 152e2c85bff341db9e4f6b7347af8034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:15.835702 2623 ts_tablet_manager.cc:1428] T d39290228be9427e9c197eb68b4347b0 P 152e2c85bff341db9e4f6b7347af8034: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:15.835703 2626 tablet_replica.cc:439] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } } }
I20260925 12:17:15.835824 2626 tablet_replica.cc:442] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:17:15.835829 2621 tablet_bootstrap.cc:492] T d39290228be9427e9c197eb68b4347b0 P 04884f65bae6405f806102a42e6344a9: Bootstrap starting.
I20260925 12:17:15.835959 2628 tablet_replica.cc:439] T d39290228be9427e9c197eb68b4347b0 P 152e2c85bff341db9e4f6b7347af8034: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } } }
I20260925 12:17:15.836055 2628 tablet_replica.cc:442] T d39290228be9427e9c197eb68b4347b0 P 152e2c85bff341db9e4f6b7347af8034: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:17:15.836457 2621 tablet_bootstrap.cc:654] T d39290228be9427e9c197eb68b4347b0 P 04884f65bae6405f806102a42e6344a9: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:15.837170 2621 tablet_bootstrap.cc:492] T d39290228be9427e9c197eb68b4347b0 P 04884f65bae6405f806102a42e6344a9: No bootstrap required, opened a new log
I20260925 12:17:15.837216 2621 ts_tablet_manager.cc:1397] T d39290228be9427e9c197eb68b4347b0 P 04884f65bae6405f806102a42e6344a9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:15.837333 2621 raft_consensus.cc:348] T d39290228be9427e9c197eb68b4347b0 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:15.837378 2621 raft_consensus.cc:374] T d39290228be9427e9c197eb68b4347b0 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:15.837389 2621 raft_consensus.cc:729] T d39290228be9427e9c197eb68b4347b0 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04884f65bae6405f806102a42e6344a9, State: Initialized, Role: FOLLOWER
I20260925 12:17:15.837438 2621 consensus_queue.cc:260] T d39290228be9427e9c197eb68b4347b0 P 04884f65bae6405f806102a42e6344a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:15.837553 2621 ts_tablet_manager.cc:1428] T d39290228be9427e9c197eb68b4347b0 P 04884f65bae6405f806102a42e6344a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:15.837579 2627 tablet_replica.cc:439] T d39290228be9427e9c197eb68b4347b0 P 04884f65bae6405f806102a42e6344a9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } } }
I20260925 12:17:15.837689 2627 tablet_replica.cc:442] T d39290228be9427e9c197eb68b4347b0 P 04884f65bae6405f806102a42e6344a9: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 12:17:15.892105 2321 tablet.cc:2369] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 12:17:15.945523 2601 tablet.cc:2369] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:17:16.228255 2626 consensus_queue.cc:1035] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:16.230479 2643 consensus_queue.cc:1035] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:16.239806 2643 raft_consensus.cc:482] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:17:16.239872 2643 raft_consensus.cc:504] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:16.240026 2643 leader_election.cc:288] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04884f65bae6405f806102a42e6344a9 (127.0.98.2:36637), 152e2c85bff341db9e4f6b7347af8034 (127.0.98.3:45487)
I20260925 12:17:16.240173 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39290228be9427e9c197eb68b4347b0" candidate_uuid: "9268de0b968a4b22a8a4306a276cdbc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04884f65bae6405f806102a42e6344a9" is_pre_election: true
I20260925 12:17:16.240175 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39290228be9427e9c197eb68b4347b0" candidate_uuid: "9268de0b968a4b22a8a4306a276cdbc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152e2c85bff341db9e4f6b7347af8034" is_pre_election: true
I20260925 12:17:16.240281 2414 raft_consensus.cc:2454] T d39290228be9427e9c197eb68b4347b0 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9268de0b968a4b22a8a4306a276cdbc5 in term 0.
I20260925 12:17:16.240280 2553 raft_consensus.cc:2454] T d39290228be9427e9c197eb68b4347b0 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9268de0b968a4b22a8a4306a276cdbc5 in term 0.
I20260925 12:17:16.240423 2206 leader_election.cc:302] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04884f65bae6405f806102a42e6344a9, 9268de0b968a4b22a8a4306a276cdbc5; no voters:
I20260925 12:17:16.240535 2643 raft_consensus.cc:2781] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:17:16.240571 2643 raft_consensus.cc:482] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:17:16.240629 2643 raft_consensus.cc:3037] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:16.241273 2643 raft_consensus.cc:504] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:16.241410 2643 leader_election.cc:288] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5 [CANDIDATE]: Term 1 election: Requested vote from peers 04884f65bae6405f806102a42e6344a9 (127.0.98.2:36637), 152e2c85bff341db9e4f6b7347af8034 (127.0.98.3:45487)
I20260925 12:17:16.241533 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39290228be9427e9c197eb68b4347b0" candidate_uuid: "9268de0b968a4b22a8a4306a276cdbc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152e2c85bff341db9e4f6b7347af8034"
I20260925 12:17:16.241534 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39290228be9427e9c197eb68b4347b0" candidate_uuid: "9268de0b968a4b22a8a4306a276cdbc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04884f65bae6405f806102a42e6344a9"
I20260925 12:17:16.241600 2414 raft_consensus.cc:3037] T d39290228be9427e9c197eb68b4347b0 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:16.241600 2553 raft_consensus.cc:3037] T d39290228be9427e9c197eb68b4347b0 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:16.242313 2414 raft_consensus.cc:2454] T d39290228be9427e9c197eb68b4347b0 P 04884f65bae6405f806102a42e6344a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9268de0b968a4b22a8a4306a276cdbc5 in term 1.
I20260925 12:17:16.242313 2553 raft_consensus.cc:2454] T d39290228be9427e9c197eb68b4347b0 P 152e2c85bff341db9e4f6b7347af8034 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9268de0b968a4b22a8a4306a276cdbc5 in term 1.
I20260925 12:17:16.242493 2206 leader_election.cc:302] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04884f65bae6405f806102a42e6344a9, 9268de0b968a4b22a8a4306a276cdbc5; no voters:
I20260925 12:17:16.242606 2643 raft_consensus.cc:2781] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:16.242687 2643 raft_consensus.cc:686] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5 [term 1 LEADER]: Becoming Leader. State: Replica: 9268de0b968a4b22a8a4306a276cdbc5, State: Running, Role: LEADER
I20260925 12:17:16.242734 2643 consensus_queue.cc:237] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:16.242980 2626 tablet_replica.cc:439] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5: TxnStatusTablet state changed. Reason: New leader 9268de0b968a4b22a8a4306a276cdbc5. Latest consensus state: current_term: 1 leader_uuid: "9268de0b968a4b22a8a4306a276cdbc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } } }
I20260925 12:17:16.243062 2626 tablet_replica.cc:442] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5: This TxnStatusTablet replica's current role is: LEADER
I20260925 12:17:16.243180 2648 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 12:17:16.243310 2648 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 12:17:16.243214 2116 catalog_manager.cc:5495] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9268de0b968a4b22a8a4306a276cdbc5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9268de0b968a4b22a8a4306a276cdbc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } health_report { overall_health: UNKNOWN } } }
I20260925 12:17:16.278159 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:16.275337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35461 (local address 127.0.0.1:56218)
0925 12:17:16.275406 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:16.275418 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:16.275461 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:16.275818 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:16.275819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:16.275820 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:17:16.275867 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:16.275869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.276619 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.276620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:16.277175 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:16.277176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.277603 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.277604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:16.277672 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:16.277684 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:17:16.277983 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:17:16.277992 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 12:17:16.278036 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 12:17:16.278093 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 12:17:16.278159 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:16.275430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56218 (local address 127.0.98.1:35461)
0925 12:17:16.275567 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:16.275569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:16.275576 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:17:16.275666 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:16.275666 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:16.275710 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:16.275812 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:16.275962 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.276521 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.277290 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.277510 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.277604 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:16.277807 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:16.277914 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:16.278006 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:16.278051 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 12:17:16.278071 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260925 12:17:16.279733 2414 raft_consensus.cc:1261] T d39290228be9427e9c197eb68b4347b0 P 04884f65bae6405f806102a42e6344a9 [term 1 FOLLOWER]: Refusing update from remote peer 9268de0b968a4b22a8a4306a276cdbc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:16.279750 2553 raft_consensus.cc:1261] T d39290228be9427e9c197eb68b4347b0 P 152e2c85bff341db9e4f6b7347af8034 [term 1 FOLLOWER]: Refusing update from remote peer 9268de0b968a4b22a8a4306a276cdbc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:16.279942 2643 consensus_queue.cc:1035] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:16.280040 2626 consensus_queue.cc:1035] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:16.280867 2627 tablet_replica.cc:439] T d39290228be9427e9c197eb68b4347b0 P 04884f65bae6405f806102a42e6344a9: TxnStatusTablet state changed. Reason: New leader 9268de0b968a4b22a8a4306a276cdbc5. Latest consensus state: current_term: 1 leader_uuid: "9268de0b968a4b22a8a4306a276cdbc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } } }
I20260925 12:17:16.280947 2627 tablet_replica.cc:442] T d39290228be9427e9c197eb68b4347b0 P 04884f65bae6405f806102a42e6344a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:17:16.280979 2628 tablet_replica.cc:439] T d39290228be9427e9c197eb68b4347b0 P 152e2c85bff341db9e4f6b7347af8034: TxnStatusTablet state changed. Reason: New leader 9268de0b968a4b22a8a4306a276cdbc5. Latest consensus state: current_term: 1 leader_uuid: "9268de0b968a4b22a8a4306a276cdbc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } } }
I20260925 12:17:16.281090 2628 tablet_replica.cc:442] T d39290228be9427e9c197eb68b4347b0 P 152e2c85bff341db9e4f6b7347af8034: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:17:16.281921 2643 tablet_replica.cc:439] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9268de0b968a4b22a8a4306a276cdbc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } } }
I20260925 12:17:16.282004 2643 tablet_replica.cc:442] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5: This TxnStatusTablet replica's current role is: LEADER
I20260925 12:17:16.282060 2643 tablet_replica.cc:439] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9268de0b968a4b22a8a4306a276cdbc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } } }
I20260925 12:17:16.282109 2643 tablet_replica.cc:442] T d39290228be9427e9c197eb68b4347b0 P 9268de0b968a4b22a8a4306a276cdbc5: This TxnStatusTablet replica's current role is: LEADER
I20260925 12:17:16.282055 2628 tablet_replica.cc:439] T d39290228be9427e9c197eb68b4347b0 P 152e2c85bff341db9e4f6b7347af8034: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9268de0b968a4b22a8a4306a276cdbc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } } }
I20260925 12:17:16.282140 2628 tablet_replica.cc:442] T d39290228be9427e9c197eb68b4347b0 P 152e2c85bff341db9e4f6b7347af8034: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:17:16.282375 2646 mvcc.cc:205] Tried to move back new op lower bound from 7333227054199693312 to 7333227054050508800. Current Snapshot: MvccSnapshot[applied={T|T < 7333227054199693312}]
I20260925 12:17:16.282665 2642 mvcc.cc:205] Tried to move back new op lower bound from 7333227054199693312 to 7333227054050508800. Current Snapshot: MvccSnapshot[applied={T|T < 7333227054199693312 or (T in {7333227054199693312})}]
I20260925 12:17:16.282895 2644 mvcc.cc:205] Tried to move back new op lower bound from 7333227054199693312 to 7333227054050508800. Current Snapshot: MvccSnapshot[applied={T|T < 7333227054199693312}]
I20260925 12:17:16.283098 2627 tablet_replica.cc:439] T d39290228be9427e9c197eb68b4347b0 P 04884f65bae6405f806102a42e6344a9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9268de0b968a4b22a8a4306a276cdbc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } } }
I20260925 12:17:16.283175 2627 tablet_replica.cc:442] T d39290228be9427e9c197eb68b4347b0 P 04884f65bae6405f806102a42e6344a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:17:16.286485 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:16.284035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.0.98.1:35461)
0925 12:17:16.284098 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:16.284100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:16.284108 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:16.284128 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:16.284129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:16.284139 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:16.284206 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:16.284298 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.285035 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.285764 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.286054 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.286125 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:16.286248 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:16.286279 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:16.286331 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:16.286376 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 12:17:16.286394 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 12:17:16.291389 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:16.288839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35461 (local address 127.0.98.1:33961)
0925 12:17:16.288961 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:16.288970 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:16.289024 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:16.289134 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:16.289134 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:16.289135 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:17:16.289185 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:16.289186 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.290015 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.290015 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:16.290591 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:16.290592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.291028 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.291029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:16.291073 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:16.291083 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:17:16.291260 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:17:16.291261 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 12:17:16.291288 (+ 27us) client_negotiation.cc:240] Negotiation successful
0925 12:17:16.291319 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":73,"threads_started":1}
I20260925 12:17:16.291392 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:16.288890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33961 (local address 127.0.98.1:35461)
0925 12:17:16.288924 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:16.288925 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:16.289032 (+ 107us) server_negotiation.cc:407] Connection header received
0925 12:17:16.289069 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:16.289070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:16.289078 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:16.289132 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:16.289219 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.289931 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.290668 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.290956 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.291023 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:16.291174 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:16.291205 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:16.291260 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:16.291292 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 12:17:16.291309 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:17:16.302846 2116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:16.302958 2116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:16.303670 2116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165: Sending DeleteTablet for 3 replicas of tablet 03e32a3e08fd498481b4419f03a38559
I20260925 12:17:16.303975 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e32a3e08fd498481b4419f03a38559 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37906
I20260925 12:17:16.304157 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e32a3e08fd498481b4419f03a38559 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56214
I20260925 12:17:16.304158 2116 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:17:16.304322 2116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:17:16.304440 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e32a3e08fd498481b4419f03a38559 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44326
I20260925 12:17:16.304620 2663 tablet_replica.cc:330] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9: stopping tablet replica
I20260925 12:17:16.304711 2663 raft_consensus.cc:2229] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:16.304767 2663 raft_consensus.cc:2258] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:16.305119 2663 ts_tablet_manager.cc:1905] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:16.305406 2534 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef076c10b5b457b93235f1207f30c93 (DEFAULT_TABLE table=test-table [id=b15156e15e224c47bc141e097cb1d678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:16.305410 2394 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef076c10b5b457b93235f1207f30c93 (DEFAULT_TABLE table=test-table [id=b15156e15e224c47bc141e097cb1d678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:16.305522 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef076c10b5b457b93235f1207f30c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:16.305526 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef076c10b5b457b93235f1207f30c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:16.306255 2665 tablet_replica.cc:330] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034: stopping tablet replica
I20260925 12:17:16.306342 2665 raft_consensus.cc:2229] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:16.306388 2665 raft_consensus.cc:2258] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:16.306797 2621 tablet_bootstrap.cc:492] T 8ef076c10b5b457b93235f1207f30c93 P 04884f65bae6405f806102a42e6344a9: Bootstrap starting.
I20260925 12:17:16.306797 2623 tablet_bootstrap.cc:492] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034: Bootstrap starting.
I20260925 12:17:16.307185 2623 tablet_bootstrap.cc:654] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:16.307185 2621 tablet_bootstrap.cc:654] T 8ef076c10b5b457b93235f1207f30c93 P 04884f65bae6405f806102a42e6344a9: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:16.307348 2665 ts_tablet_manager.cc:1905] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:16.307725 2254 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef076c10b5b457b93235f1207f30c93 (DEFAULT_TABLE table=test-table [id=b15156e15e224c47bc141e097cb1d678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:16.307778 2663 ts_tablet_manager.cc:1918] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:17:16.307837 2663 log.cc:1196] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/03e32a3e08fd498481b4419f03a38559
I20260925 12:17:16.307832 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef076c10b5b457b93235f1207f30c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:16.308089 2663 ts_tablet_manager.cc:1939] T 03e32a3e08fd498481b4419f03a38559 P 04884f65bae6405f806102a42e6344a9: Deleting consensus metadata
I20260925 12:17:16.308918 2620 tablet_bootstrap.cc:492] T 8ef076c10b5b457b93235f1207f30c93 P 9268de0b968a4b22a8a4306a276cdbc5: Bootstrap starting.
I20260925 12:17:16.309219 2620 tablet_bootstrap.cc:654] T 8ef076c10b5b457b93235f1207f30c93 P 9268de0b968a4b22a8a4306a276cdbc5: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:16.309366 2665 ts_tablet_manager.cc:1918] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:17:16.309389 2621 tablet_bootstrap.cc:492] T 8ef076c10b5b457b93235f1207f30c93 P 04884f65bae6405f806102a42e6344a9: No bootstrap required, opened a new log
I20260925 12:17:16.309422 2665 log.cc:1196] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/03e32a3e08fd498481b4419f03a38559
I20260925 12:17:16.309430 2621 ts_tablet_manager.cc:1397] T 8ef076c10b5b457b93235f1207f30c93 P 04884f65bae6405f806102a42e6344a9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 12:17:16.309607 2621 raft_consensus.cc:348] T 8ef076c10b5b457b93235f1207f30c93 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:16.309695 2621 raft_consensus.cc:374] T 8ef076c10b5b457b93235f1207f30c93 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:16.309700 2665 ts_tablet_manager.cc:1939] T 03e32a3e08fd498481b4419f03a38559 P 152e2c85bff341db9e4f6b7347af8034: Deleting consensus metadata
I20260925 12:17:16.309726 2621 raft_consensus.cc:729] T 8ef076c10b5b457b93235f1207f30c93 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04884f65bae6405f806102a42e6344a9, State: Initialized, Role: FOLLOWER
I20260925 12:17:16.309778 2621 consensus_queue.cc:260] T 8ef076c10b5b457b93235f1207f30c93 P 04884f65bae6405f806102a42e6344a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:16.309887 2621 ts_tablet_manager.cc:1428] T 8ef076c10b5b457b93235f1207f30c93 P 04884f65bae6405f806102a42e6344a9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:16.310066 2101 catalog_manager.cc:4841] TS 152e2c85bff341db9e4f6b7347af8034 (127.0.98.3:45487): tablet 03e32a3e08fd498481b4419f03a38559 (table test-table [id=b260325130fa45f49f2a8bfbd6806f5c]) successfully deleted
I20260925 12:17:16.310293 2620 tablet_bootstrap.cc:492] T 8ef076c10b5b457b93235f1207f30c93 P 9268de0b968a4b22a8a4306a276cdbc5: No bootstrap required, opened a new log
I20260925 12:17:16.310336 2620 ts_tablet_manager.cc:1397] T 8ef076c10b5b457b93235f1207f30c93 P 9268de0b968a4b22a8a4306a276cdbc5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 12:17:16.310346 2623 tablet_bootstrap.cc:492] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034: No bootstrap required, opened a new log
I20260925 12:17:16.310382 2623 ts_tablet_manager.cc:1397] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 12:17:16.310431 2620 raft_consensus.cc:348] T 8ef076c10b5b457b93235f1207f30c93 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:16.310487 2620 raft_consensus.cc:374] T 8ef076c10b5b457b93235f1207f30c93 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:16.310509 2620 raft_consensus.cc:729] T 8ef076c10b5b457b93235f1207f30c93 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9268de0b968a4b22a8a4306a276cdbc5, State: Initialized, Role: FOLLOWER
I20260925 12:17:16.310495 2623 raft_consensus.cc:348] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:16.310554 2623 raft_consensus.cc:374] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:16.310545 2620 consensus_queue.cc:260] T 8ef076c10b5b457b93235f1207f30c93 P 9268de0b968a4b22a8a4306a276cdbc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:16.310573 2623 raft_consensus.cc:729] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152e2c85bff341db9e4f6b7347af8034, State: Initialized, Role: FOLLOWER
I20260925 12:17:16.310631 2620 ts_tablet_manager.cc:1428] T 8ef076c10b5b457b93235f1207f30c93 P 9268de0b968a4b22a8a4306a276cdbc5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:16.310628 2623 consensus_queue.cc:260] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:16.310657 2102 catalog_manager.cc:4841] TS 04884f65bae6405f806102a42e6344a9 (127.0.98.2:36637): tablet 03e32a3e08fd498481b4419f03a38559 (table test-table [id=b260325130fa45f49f2a8bfbd6806f5c]) successfully deleted
I20260925 12:17:16.310714 2623 ts_tablet_manager.cc:1428] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:16.311733 2664 tablet_replica.cc:330] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5: stopping tablet replica
I20260925 12:17:16.311807 2664 raft_consensus.cc:2229] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:17:16.311923 2664 raft_consensus.cc:2258] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:16.312232 2664 ts_tablet_manager.cc:1905] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:16.313588 2664 ts_tablet_manager.cc:1918] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:17:16.313649 2664 log.cc:1196] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-0/wal/wals/03e32a3e08fd498481b4419f03a38559
I20260925 12:17:16.313879 2664 ts_tablet_manager.cc:1939] T 03e32a3e08fd498481b4419f03a38559 P 9268de0b968a4b22a8a4306a276cdbc5: Deleting consensus metadata
I20260925 12:17:16.314209 2103 catalog_manager.cc:4841] TS 9268de0b968a4b22a8a4306a276cdbc5 (127.0.98.1:35461): tablet 03e32a3e08fd498481b4419f03a38559 (table test-table [id=b260325130fa45f49f2a8bfbd6806f5c]) successfully deleted
I20260925 12:17:16.353471 2628 raft_consensus.cc:482] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:17:16.353551 2628 raft_consensus.cc:504] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:16.353886 2628 leader_election.cc:288] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9268de0b968a4b22a8a4306a276cdbc5 (127.0.98.1:35461), 04884f65bae6405f806102a42e6344a9 (127.0.98.2:36637)
I20260925 12:17:16.357158 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:16.354048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36637 (local address 127.0.98.3:48361)
0925 12:17:16.354178 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:16.354188 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:16.354232 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:16.354480 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:16.354481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:16.354483 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:16.354558 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:16.354560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.355435 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.355436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:16.356244 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:16.356246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.356893 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.356895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:16.356972 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:16.356977 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:17:16.357038 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 12:17:16.357074 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":49,"threads_started":1}
I20260925 12:17:16.357229 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:16.354197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48361 (local address 127.0.98.2:36637)
0925 12:17:16.354331 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:16.354333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:16.354341 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:16.354363 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:16.354364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:16.354372 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:16.354442 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:16.354652 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.355340 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.356347 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.356808 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.356877 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:16.356889 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:16.357067 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 12:17:16.357115 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":48,"threads_started":1}
I20260925 12:17:16.357630 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:16.354046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35461 (local address 127.0.98.3:41043)
0925 12:17:16.354162 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:16.354173 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:16.354248 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:16.354429 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:16.354430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:16.354432 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:16.354499 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:16.354501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.355405 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.355406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:16.356585 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:16.356587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.357356 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.357357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:16.357430 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:16.357434 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:17:16.357492 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 12:17:16.357540 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":24,"thread_start_us":60,"threads_started":1}
I20260925 12:17:16.357715 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef076c10b5b457b93235f1207f30c93" candidate_uuid: "152e2c85bff341db9e4f6b7347af8034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04884f65bae6405f806102a42e6344a9" is_pre_election: true
I20260925 12:17:16.357812 2414 raft_consensus.cc:2454] T 8ef076c10b5b457b93235f1207f30c93 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 152e2c85bff341db9e4f6b7347af8034 in term 0.
I20260925 12:17:16.357827 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:16.354100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41043 (local address 127.0.98.1:35461)
0925 12:17:16.354150 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:16.354153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:16.354226 (+ 73us) server_negotiation.cc:407] Connection header received
0925 12:17:16.354322 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:16.354323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:16.354333 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:16.354362 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:16.354571 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.355306 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.356699 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.357241 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.357311 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:16.357321 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:16.357527 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 12:17:16.357564 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 12:17:16.357998 2486 leader_election.cc:302] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04884f65bae6405f806102a42e6344a9, 152e2c85bff341db9e4f6b7347af8034; no voters:
I20260925 12:17:16.358135 2628 raft_consensus.cc:2781] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:17:16.358174 2628 raft_consensus.cc:482] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:17:16.358222 2628 raft_consensus.cc:3037] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:16.358834 2628 raft_consensus.cc:504] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:16.358955 2628 leader_election.cc:288] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [CANDIDATE]: Term 1 election: Requested vote from peers 9268de0b968a4b22a8a4306a276cdbc5 (127.0.98.1:35461), 04884f65bae6405f806102a42e6344a9 (127.0.98.2:36637)
I20260925 12:17:16.358067 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef076c10b5b457b93235f1207f30c93" candidate_uuid: "152e2c85bff341db9e4f6b7347af8034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9268de0b968a4b22a8a4306a276cdbc5" is_pre_election: true
I20260925 12:17:16.359090 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef076c10b5b457b93235f1207f30c93" candidate_uuid: "152e2c85bff341db9e4f6b7347af8034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9268de0b968a4b22a8a4306a276cdbc5"
I20260925 12:17:16.359238 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef076c10b5b457b93235f1207f30c93" candidate_uuid: "152e2c85bff341db9e4f6b7347af8034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04884f65bae6405f806102a42e6344a9"
I20260925 12:17:16.359095 2274 raft_consensus.cc:2454] T 8ef076c10b5b457b93235f1207f30c93 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 152e2c85bff341db9e4f6b7347af8034 in term 0.
I20260925 12:17:16.359301 2414 raft_consensus.cc:3037] T 8ef076c10b5b457b93235f1207f30c93 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:16.359934 2414 raft_consensus.cc:2454] T 8ef076c10b5b457b93235f1207f30c93 P 04884f65bae6405f806102a42e6344a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 152e2c85bff341db9e4f6b7347af8034 in term 1.
I20260925 12:17:16.360083 2486 leader_election.cc:302] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04884f65bae6405f806102a42e6344a9, 152e2c85bff341db9e4f6b7347af8034; no voters: 9268de0b968a4b22a8a4306a276cdbc5
I20260925 12:17:16.360188 2628 raft_consensus.cc:2781] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:16.360317 2628 raft_consensus.cc:686] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [term 1 LEADER]: Becoming Leader. State: Replica: 152e2c85bff341db9e4f6b7347af8034, State: Running, Role: LEADER
I20260925 12:17:16.360384 2628 consensus_queue.cc:237] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:16.360989 2116 catalog_manager.cc:5495] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 reported cstate change: term changed from 0 to 1, leader changed from <none> to 152e2c85bff341db9e4f6b7347af8034 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "152e2c85bff341db9e4f6b7347af8034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } health_report { overall_health: HEALTHY } } }
I20260925 12:17:16.369917 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:16.367660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.0.98.3:45487)
0925 12:17:16.367794 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:16.367796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:16.367802 (+ 6us) server_negotiation.cc:407] Connection header received
0925 12:17:16.367836 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:16.367837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:16.367850 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:16.367932 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:16.368024 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.368485 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.369162 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.369365 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.369434 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:16.369606 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:16.369705 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:16.369782 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:16.369805 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 12:17:16.369822 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260925 12:17:16.374591 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:16.372412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56885 (local address 127.0.98.1:35461)
0925 12:17:16.372443 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:16.372444 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:16.372585 (+ 141us) server_negotiation.cc:407] Connection header received
0925 12:17:16.372646 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:16.372647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:16.372657 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:16.372705 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:16.372795 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.373198 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.373930 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.374124 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.374203 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:16.374370 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:16.374405 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:16.374464 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:16.374512 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 12:17:16.374530 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 12:17:16.374666 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:16.372354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35461 (local address 127.0.98.3:56885)
0925 12:17:16.372510 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:16.372518 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:16.372603 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:16.372708 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:16.372708 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:16.372709 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:17:16.372754 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:16.372755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.373281 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.373282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:16.373837 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:16.373839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.374206 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.374207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:16.374265 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:16.374279 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:17:16.374465 (+ 186us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:17:16.374467 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:17:16.374511 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 12:17:16.374547 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260925 12:17:16.377038 2414 raft_consensus.cc:1261] T 8ef076c10b5b457b93235f1207f30c93 P 04884f65bae6405f806102a42e6344a9 [term 1 FOLLOWER]: Refusing update from remote peer 152e2c85bff341db9e4f6b7347af8034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:16.377120 2274 raft_consensus.cc:3037] T 8ef076c10b5b457b93235f1207f30c93 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:16.377229 2672 consensus_queue.cc:1035] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:16.377990 2274 raft_consensus.cc:1261] T 8ef076c10b5b457b93235f1207f30c93 P 9268de0b968a4b22a8a4306a276cdbc5 [term 1 FOLLOWER]: Refusing update from remote peer 152e2c85bff341db9e4f6b7347af8034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:16.378206 2628 consensus_queue.cc:1035] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:16.384356 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:16.382157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45487 (local address 127.0.98.1:35893)
0925 12:17:16.382205 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:16.382216 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:16.382268 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:16.382380 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:16.382381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:16.382382 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:17:16.382410 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:16.382411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.383055 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.383055 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:16.383566 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:16.383568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.383982 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.383983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:16.384030 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:16.384036 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:17:16.384175 (+ 139us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:17:16.384177 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:17:16.384232 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 12:17:16.384294 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 12:17:16.384596 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:16.382248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35893 (local address 127.0.98.3:45487)
0925 12:17:16.382288 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:16.382290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:16.382309 (+ 19us) server_negotiation.cc:407] Connection header received
0925 12:17:16.382328 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:16.382329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:16.382337 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:16.382380 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:16.382468 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.382966 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.383672 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:16.383890 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:16.383949 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:16.384099 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:16.384123 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:16.384158 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:16.384522 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 12:17:16.384541 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 12:17:16.384816 2646 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 12:17:17.903828 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:17.903918 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:17.904709 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165: Sending DeleteTablet for 3 replicas of tablet 8ef076c10b5b457b93235f1207f30c93
I20260925 12:17:17.904956 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef076c10b5b457b93235f1207f30c93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44326
I20260925 12:17:17.904985 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef076c10b5b457b93235f1207f30c93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56214
I20260925 12:17:17.905139 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef076c10b5b457b93235f1207f30c93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37906
I20260925 12:17:17.905273 2704 tablet_replica.cc:330] T 8ef076c10b5b457b93235f1207f30c93 P 9268de0b968a4b22a8a4306a276cdbc5: stopping tablet replica
I20260925 12:17:17.905332 2704 raft_consensus.cc:2229] T 8ef076c10b5b457b93235f1207f30c93 P 9268de0b968a4b22a8a4306a276cdbc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:17.905373 2704 raft_consensus.cc:2258] T 8ef076c10b5b457b93235f1207f30c93 P 9268de0b968a4b22a8a4306a276cdbc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:17.905458 2705 tablet_replica.cc:330] T 8ef076c10b5b457b93235f1207f30c93 P 04884f65bae6405f806102a42e6344a9: stopping tablet replica
I20260925 12:17:17.905539 2705 raft_consensus.cc:2229] T 8ef076c10b5b457b93235f1207f30c93 P 04884f65bae6405f806102a42e6344a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:17.905596 2705 raft_consensus.cc:2258] T 8ef076c10b5b457b93235f1207f30c93 P 04884f65bae6405f806102a42e6344a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:17.905686 2704 ts_tablet_manager.cc:1905] T 8ef076c10b5b457b93235f1207f30c93 P 9268de0b968a4b22a8a4306a276cdbc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:17.905848 2705 ts_tablet_manager.cc:1905] T 8ef076c10b5b457b93235f1207f30c93 P 04884f65bae6405f806102a42e6344a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:17.908102 2704 ts_tablet_manager.cc:1918] T 8ef076c10b5b457b93235f1207f30c93 P 9268de0b968a4b22a8a4306a276cdbc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 12:17:17.908186 2704 log.cc:1196] T 8ef076c10b5b457b93235f1207f30c93 P 9268de0b968a4b22a8a4306a276cdbc5: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-0/wal/wals/8ef076c10b5b457b93235f1207f30c93
I20260925 12:17:17.908396 2704 ts_tablet_manager.cc:1939] T 8ef076c10b5b457b93235f1207f30c93 P 9268de0b968a4b22a8a4306a276cdbc5: Deleting consensus metadata
I20260925 12:17:17.908552 2703 tablet_replica.cc:330] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034: stopping tablet replica
I20260925 12:17:17.908629 2703 raft_consensus.cc:2229] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:17:17.908730 2703 raft_consensus.cc:2258] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:17.908874 2705 ts_tablet_manager.cc:1918] T 8ef076c10b5b457b93235f1207f30c93 P 04884f65bae6405f806102a42e6344a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 12:17:17.908932 2705 log.cc:1196] T 8ef076c10b5b457b93235f1207f30c93 P 04884f65bae6405f806102a42e6344a9: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/8ef076c10b5b457b93235f1207f30c93
I20260925 12:17:17.908960 2103 catalog_manager.cc:4841] TS 9268de0b968a4b22a8a4306a276cdbc5 (127.0.98.1:35461): tablet 8ef076c10b5b457b93235f1207f30c93 (table test-table [id=b15156e15e224c47bc141e097cb1d678]) successfully deleted
I20260925 12:17:17.909190 2705 ts_tablet_manager.cc:1939] T 8ef076c10b5b457b93235f1207f30c93 P 04884f65bae6405f806102a42e6344a9: Deleting consensus metadata
I20260925 12:17:17.909472 2102 catalog_manager.cc:4841] TS 04884f65bae6405f806102a42e6344a9 (127.0.98.2:36637): tablet 8ef076c10b5b457b93235f1207f30c93 (table test-table [id=b15156e15e224c47bc141e097cb1d678]) successfully deleted
I20260925 12:17:17.910794 2703 ts_tablet_manager.cc:1905] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:17.912076 2703 ts_tablet_manager.cc:1918] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 12:17:17.912153 2703 log.cc:1196] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/8ef076c10b5b457b93235f1207f30c93
I20260925 12:17:17.912372 2703 ts_tablet_manager.cc:1939] T 8ef076c10b5b457b93235f1207f30c93 P 152e2c85bff341db9e4f6b7347af8034: Deleting consensus metadata
I20260925 12:17:17.912720 2101 catalog_manager.cc:4841] TS 152e2c85bff341db9e4f6b7347af8034 (127.0.98.3:45487): tablet 8ef076c10b5b457b93235f1207f30c93 (table test-table [id=b15156e15e224c47bc141e097cb1d678]) successfully deleted
Sep 25 12:17:17 dist-test-slave-clmx krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338637, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 12:17:17 dist-test-slave-clmx krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338637, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:17.943938 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:17.922408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.0.98.62:36621)
0925 12:17:17.922558 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:17.922559 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:17.922567 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:17.922585 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:17.922586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:17.922595 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:17.922651 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:17.923143 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:17.924035 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:17.924821 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:17.925038 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:17.925142 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:17.941491 (+ 16349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:17.941493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:17.941493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:17.941501 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:17.942759 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:17.943122 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:17.943123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:17.943124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:17.943181 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:17.943376 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:17.943377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:17.943377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:17.943491 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:17.943646 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:17.943739 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 12:17:17.943824 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260925 12:17:17.945451 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:17:17.945593 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:17:17.946828 2254 tablet_service.cc:1467] Processing CreateTablet for tablet e3b7fe7e341a412d951d3bbd03a3ab28 (DEFAULT_TABLE table=test-table [id=dcbe26dd11c94d00862018f37c2b06ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:17.946852 2394 tablet_service.cc:1467] Processing CreateTablet for tablet e3b7fe7e341a412d951d3bbd03a3ab28 (DEFAULT_TABLE table=test-table [id=dcbe26dd11c94d00862018f37c2b06ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:17.946848 2534 tablet_service.cc:1467] Processing CreateTablet for tablet e3b7fe7e341a412d951d3bbd03a3ab28 (DEFAULT_TABLE table=test-table [id=dcbe26dd11c94d00862018f37c2b06ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:17.946971 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b7fe7e341a412d951d3bbd03a3ab28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:17.946971 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b7fe7e341a412d951d3bbd03a3ab28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:17.947002 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b7fe7e341a412d951d3bbd03a3ab28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:17.948138 2714 tablet_bootstrap.cc:492] T e3b7fe7e341a412d951d3bbd03a3ab28 P 152e2c85bff341db9e4f6b7347af8034: Bootstrap starting.
I20260925 12:17:17.948297 2716 tablet_bootstrap.cc:492] T e3b7fe7e341a412d951d3bbd03a3ab28 P 9268de0b968a4b22a8a4306a276cdbc5: Bootstrap starting.
I20260925 12:17:17.948482 2714 tablet_bootstrap.cc:654] T e3b7fe7e341a412d951d3bbd03a3ab28 P 152e2c85bff341db9e4f6b7347af8034: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:17.948621 2716 tablet_bootstrap.cc:654] T e3b7fe7e341a412d951d3bbd03a3ab28 P 9268de0b968a4b22a8a4306a276cdbc5: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:17.948907 2715 tablet_bootstrap.cc:492] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9: Bootstrap starting.
I20260925 12:17:17.949061 2714 tablet_bootstrap.cc:492] T e3b7fe7e341a412d951d3bbd03a3ab28 P 152e2c85bff341db9e4f6b7347af8034: No bootstrap required, opened a new log
I20260925 12:17:17.949110 2714 ts_tablet_manager.cc:1397] T e3b7fe7e341a412d951d3bbd03a3ab28 P 152e2c85bff341db9e4f6b7347af8034: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:17.949169 2716 tablet_bootstrap.cc:492] T e3b7fe7e341a412d951d3bbd03a3ab28 P 9268de0b968a4b22a8a4306a276cdbc5: No bootstrap required, opened a new log
I20260925 12:17:17.949216 2716 ts_tablet_manager.cc:1397] T e3b7fe7e341a412d951d3bbd03a3ab28 P 9268de0b968a4b22a8a4306a276cdbc5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:17.949216 2714 raft_consensus.cc:348] T e3b7fe7e341a412d951d3bbd03a3ab28 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:17.949283 2714 raft_consensus.cc:374] T e3b7fe7e341a412d951d3bbd03a3ab28 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:17.949309 2714 raft_consensus.cc:729] T e3b7fe7e341a412d951d3bbd03a3ab28 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152e2c85bff341db9e4f6b7347af8034, State: Initialized, Role: FOLLOWER
I20260925 12:17:17.949358 2714 consensus_queue.cc:260] T e3b7fe7e341a412d951d3bbd03a3ab28 P 152e2c85bff341db9e4f6b7347af8034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:17.949347 2716 raft_consensus.cc:348] T e3b7fe7e341a412d951d3bbd03a3ab28 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:17.949411 2716 raft_consensus.cc:374] T e3b7fe7e341a412d951d3bbd03a3ab28 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:17.949445 2716 raft_consensus.cc:729] T e3b7fe7e341a412d951d3bbd03a3ab28 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9268de0b968a4b22a8a4306a276cdbc5, State: Initialized, Role: FOLLOWER
I20260925 12:17:17.949499 2716 consensus_queue.cc:260] T e3b7fe7e341a412d951d3bbd03a3ab28 P 9268de0b968a4b22a8a4306a276cdbc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:17.949522 2714 ts_tablet_manager.cc:1428] T e3b7fe7e341a412d951d3bbd03a3ab28 P 152e2c85bff341db9e4f6b7347af8034: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:17.949595 2716 ts_tablet_manager.cc:1428] T e3b7fe7e341a412d951d3bbd03a3ab28 P 9268de0b968a4b22a8a4306a276cdbc5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:17.949940 2715 tablet_bootstrap.cc:654] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:17.950502 2715 tablet_bootstrap.cc:492] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9: No bootstrap required, opened a new log
I20260925 12:17:17.950546 2715 ts_tablet_manager.cc:1397] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:17:17.950706 2715 raft_consensus.cc:348] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:17.950771 2715 raft_consensus.cc:374] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:17.950805 2715 raft_consensus.cc:729] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04884f65bae6405f806102a42e6344a9, State: Initialized, Role: FOLLOWER
I20260925 12:17:17.950855 2715 consensus_queue.cc:260] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:17.951014 2715 ts_tablet_manager.cc:1428] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:18.147842 2720 raft_consensus.cc:482] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:17:18.147962 2720 raft_consensus.cc:504] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:18.148355 2720 leader_election.cc:288] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9268de0b968a4b22a8a4306a276cdbc5 (127.0.98.1:35461), 152e2c85bff341db9e4f6b7347af8034 (127.0.98.3:45487)
I20260925 12:17:18.151430 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:18.148496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45487 (local address 127.0.98.2:54145)
0925 12:17:18.148716 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:18.148726 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:18.148781 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:18.148940 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:18.148941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:18.148944 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:18.149045 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:18.149047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:18.149739 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:18.149740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:18.150559 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:18.150561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:18.151098 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:18.151098 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:18.151259 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:18.151264 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:17:18.151321 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 12:17:18.151351 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":27,"thread_start_us":112,"threads_started":1}
I20260925 12:17:18.151459 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:18.148544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54145 (local address 127.0.98.3:45487)
0925 12:17:18.148710 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:18.148712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:18.148774 (+ 62us) server_negotiation.cc:407] Connection header received
0925 12:17:18.148831 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:18.148832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:18.148842 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:18.148926 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:18.149135 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:18.149662 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:18.150649 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:18.151009 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:18.151099 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:18.151110 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:18.151322 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 12:17:18.151368 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":71,"threads_started":1}
I20260925 12:17:18.151705 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b7fe7e341a412d951d3bbd03a3ab28" candidate_uuid: "04884f65bae6405f806102a42e6344a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152e2c85bff341db9e4f6b7347af8034" is_pre_election: true
I20260925 12:17:18.151793 2553 raft_consensus.cc:2454] T e3b7fe7e341a412d951d3bbd03a3ab28 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04884f65bae6405f806102a42e6344a9 in term 0.
I20260925 12:17:18.151818 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:18.148559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59585 (local address 127.0.98.1:35461)
0925 12:17:18.148734 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:18.148736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:18.148847 (+ 111us) server_negotiation.cc:407] Connection header received
0925 12:17:18.148899 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:18.148900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:18.148925 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:18.149013 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:18.149222 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:18.149679 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:18.150667 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:18.151011 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:18.151099 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:18.151111 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:18.151710 (+ 599us) server_negotiation.cc:299] Negotiation successful
0925 12:17:18.151753 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":67,"threads_started":1}
I20260925 12:17:18.151836 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:18.148495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35461 (local address 127.0.98.2:59585)
0925 12:17:18.148760 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:18.148770 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:18.148837 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:18.149052 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:18.149052 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:18.149054 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:18.149151 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:18.149153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:18.149788 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:18.149793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:18.150558 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:18.150560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:18.151100 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:18.151101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:18.151649 (+ 548us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:18.151654 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:17:18.151706 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 12:17:18.151748 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":2,"thread_start_us":124,"threads_started":1}
I20260925 12:17:18.152045 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b7fe7e341a412d951d3bbd03a3ab28" candidate_uuid: "04884f65bae6405f806102a42e6344a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9268de0b968a4b22a8a4306a276cdbc5" is_pre_election: true
I20260925 12:17:18.152137 2274 raft_consensus.cc:2454] T e3b7fe7e341a412d951d3bbd03a3ab28 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04884f65bae6405f806102a42e6344a9 in term 0.
I20260925 12:17:18.152076 2345 leader_election.cc:302] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04884f65bae6405f806102a42e6344a9, 152e2c85bff341db9e4f6b7347af8034; no voters:
I20260925 12:17:18.152210 2720 raft_consensus.cc:2781] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:17:18.152249 2720 raft_consensus.cc:482] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:17:18.152267 2720 raft_consensus.cc:3037] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:18.152993 2720 raft_consensus.cc:504] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:18.153126 2720 leader_election.cc:288] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [CANDIDATE]: Term 1 election: Requested vote from peers 9268de0b968a4b22a8a4306a276cdbc5 (127.0.98.1:35461), 152e2c85bff341db9e4f6b7347af8034 (127.0.98.3:45487)
I20260925 12:17:18.153254 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b7fe7e341a412d951d3bbd03a3ab28" candidate_uuid: "04884f65bae6405f806102a42e6344a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9268de0b968a4b22a8a4306a276cdbc5"
I20260925 12:17:18.153283 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b7fe7e341a412d951d3bbd03a3ab28" candidate_uuid: "04884f65bae6405f806102a42e6344a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152e2c85bff341db9e4f6b7347af8034"
I20260925 12:17:18.153333 2274 raft_consensus.cc:3037] T e3b7fe7e341a412d951d3bbd03a3ab28 P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:18.153347 2553 raft_consensus.cc:3037] T e3b7fe7e341a412d951d3bbd03a3ab28 P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:18.153877 2553 raft_consensus.cc:2454] T e3b7fe7e341a412d951d3bbd03a3ab28 P 152e2c85bff341db9e4f6b7347af8034 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04884f65bae6405f806102a42e6344a9 in term 1.
I20260925 12:17:18.153954 2274 raft_consensus.cc:2454] T e3b7fe7e341a412d951d3bbd03a3ab28 P 9268de0b968a4b22a8a4306a276cdbc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04884f65bae6405f806102a42e6344a9 in term 1.
I20260925 12:17:18.154058 2345 leader_election.cc:302] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04884f65bae6405f806102a42e6344a9, 152e2c85bff341db9e4f6b7347af8034; no voters:
I20260925 12:17:18.154136 2720 raft_consensus.cc:2781] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:18.154250 2720 raft_consensus.cc:686] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [term 1 LEADER]: Becoming Leader. State: Replica: 04884f65bae6405f806102a42e6344a9, State: Running, Role: LEADER
I20260925 12:17:18.154333 2720 consensus_queue.cc:237] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:18.154965 2117 catalog_manager.cc:5495] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04884f65bae6405f806102a42e6344a9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "04884f65bae6405f806102a42e6344a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } health_report { overall_health: UNKNOWN } } }
I20260925 12:17:18.177431 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:18.175190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37910 (local address 127.0.98.2:36637)
0925 12:17:18.175332 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:18.175334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:18.175341 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:17:18.175377 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:18.175378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:18.175394 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:18.175462 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:18.175569 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:18.176078 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:18.176782 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:18.176978 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:18.177026 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:18.177161 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:18.177258 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:18.177310 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:18.177350 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 12:17:18.177363 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260925 12:17:18.181514 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:18.179753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38323 (local address 127.0.98.1:35461)
0925 12:17:18.179783 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:18.179785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:18.179797 (+ 12us) server_negotiation.cc:407] Connection header received
0925 12:17:18.179839 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:18.179840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:18.179847 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:18.179892 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:18.179968 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:18.180350 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:18.180964 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:18.181122 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:18.181172 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:18.181341 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:18.181363 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:18.181400 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:18.181454 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 12:17:18.181469 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 12:17:18.181541 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:18.179585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35461 (local address 127.0.98.2:38323)
0925 12:17:18.179745 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:18.179754 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:18.179805 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:18.179895 (+ 90us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:18.179896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:18.179897 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:17:18.179937 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:18.179938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:18.180405 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:18.180406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:18.180901 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:18.180903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:18.181214 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:18.181215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:18.181264 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:18.181275 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:17:18.181418 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:17:18.181419 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 12:17:18.181453 (+ 34us) client_negotiation.cc:240] Negotiation successful
0925 12:17:18.181481 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":106,"threads_started":1}
I20260925 12:17:18.183415 2274 raft_consensus.cc:1261] T e3b7fe7e341a412d951d3bbd03a3ab28 P 9268de0b968a4b22a8a4306a276cdbc5 [term 1 FOLLOWER]: Refusing update from remote peer 04884f65bae6405f806102a42e6344a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:18.183385 2553 raft_consensus.cc:1261] T e3b7fe7e341a412d951d3bbd03a3ab28 P 152e2c85bff341db9e4f6b7347af8034 [term 1 FOLLOWER]: Refusing update from remote peer 04884f65bae6405f806102a42e6344a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:18.183772 2720 consensus_queue.cc:1035] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:18.183919 2725 consensus_queue.cc:1035] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:18.190189 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:18.188108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41227 (local address 127.0.98.2:36637)
0925 12:17:18.188191 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:18.188193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:18.188269 (+ 76us) server_negotiation.cc:407] Connection header received
0925 12:17:18.188350 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:18.188351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:18.188359 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:18.188396 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:18.188482 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:18.188891 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:18.189557 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:18.189716 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:18.189797 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:18.189946 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:18.189975 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:18.190020 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:18.190117 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 12:17:18.190134 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 12:17:18.190201 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:18.188034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36637 (local address 127.0.98.1:41227)
0925 12:17:18.188214 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:18.188225 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:18.188283 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:18.188397 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:18.188398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:18.188399 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:17:18.188448 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:18.188449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:18.188954 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:18.188954 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:18.189475 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:18.189477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:18.189818 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:18.189819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:18.189871 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:18.189876 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:17:18.190025 (+ 149us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:17:18.190027 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:17:18.190076 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 12:17:18.190127 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":97,"threads_started":1}
I20260925 12:17:18.196679 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:18.196762 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:18.197377 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165: Sending DeleteTablet for 3 replicas of tablet e3b7fe7e341a412d951d3bbd03a3ab28
I20260925 12:17:18.197659 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b7fe7e341a412d951d3bbd03a3ab28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44326
I20260925 12:17:18.197672 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b7fe7e341a412d951d3bbd03a3ab28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56214
I20260925 12:17:18.197708 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b7fe7e341a412d951d3bbd03a3ab28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37906
I20260925 12:17:18.197758 2703 tablet_replica.cc:330] T e3b7fe7e341a412d951d3bbd03a3ab28 P 152e2c85bff341db9e4f6b7347af8034: stopping tablet replica
I20260925 12:17:18.197778 2705 tablet_replica.cc:330] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9: stopping tablet replica
I20260925 12:17:18.197814 2705 raft_consensus.cc:2229] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:17:18.197814 2703 raft_consensus.cc:2229] T e3b7fe7e341a412d951d3bbd03a3ab28 P 152e2c85bff341db9e4f6b7347af8034 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:18.197845 2703 raft_consensus.cc:2258] T e3b7fe7e341a412d951d3bbd03a3ab28 P 152e2c85bff341db9e4f6b7347af8034 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:18.197887 2705 raft_consensus.cc:2258] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:18.197901 2704 tablet_replica.cc:330] T e3b7fe7e341a412d951d3bbd03a3ab28 P 9268de0b968a4b22a8a4306a276cdbc5: stopping tablet replica
I20260925 12:17:18.197952 2704 raft_consensus.cc:2229] T e3b7fe7e341a412d951d3bbd03a3ab28 P 9268de0b968a4b22a8a4306a276cdbc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:18.197996 2704 raft_consensus.cc:2258] T e3b7fe7e341a412d951d3bbd03a3ab28 P 9268de0b968a4b22a8a4306a276cdbc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:18.198200 2705 ts_tablet_manager.cc:1905] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:18.198318 2703 ts_tablet_manager.cc:1905] T e3b7fe7e341a412d951d3bbd03a3ab28 P 152e2c85bff341db9e4f6b7347af8034: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:18.199798 2704 ts_tablet_manager.cc:1905] T e3b7fe7e341a412d951d3bbd03a3ab28 P 9268de0b968a4b22a8a4306a276cdbc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:18.199914 2703 ts_tablet_manager.cc:1918] T e3b7fe7e341a412d951d3bbd03a3ab28 P 152e2c85bff341db9e4f6b7347af8034: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:17:18.199960 2703 log.cc:1196] T e3b7fe7e341a412d951d3bbd03a3ab28 P 152e2c85bff341db9e4f6b7347af8034: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/e3b7fe7e341a412d951d3bbd03a3ab28
I20260925 12:17:18.199903 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:17:18.200022 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:17:18.200188 2703 ts_tablet_manager.cc:1939] T e3b7fe7e341a412d951d3bbd03a3ab28 P 152e2c85bff341db9e4f6b7347af8034: Deleting consensus metadata
I20260925 12:17:18.200577 2101 catalog_manager.cc:4841] TS 152e2c85bff341db9e4f6b7347af8034 (127.0.98.3:45487): tablet e3b7fe7e341a412d951d3bbd03a3ab28 (table test-table [id=dcbe26dd11c94d00862018f37c2b06ef]) successfully deleted
I20260925 12:17:18.201136 2534 tablet_service.cc:1467] Processing CreateTablet for tablet c6bfe727714e4aabb31fd13b45cb217f (DEFAULT_TABLE table=test-table [id=a9dd70701382449b94315db6ed98422b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:18.201139 2254 tablet_service.cc:1467] Processing CreateTablet for tablet c6bfe727714e4aabb31fd13b45cb217f (DEFAULT_TABLE table=test-table [id=a9dd70701382449b94315db6ed98422b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:18.201257 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6bfe727714e4aabb31fd13b45cb217f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:18.201259 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6bfe727714e4aabb31fd13b45cb217f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:18.201356 2704 ts_tablet_manager.cc:1918] T e3b7fe7e341a412d951d3bbd03a3ab28 P 9268de0b968a4b22a8a4306a276cdbc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:17:18.201401 2704 log.cc:1196] T e3b7fe7e341a412d951d3bbd03a3ab28 P 9268de0b968a4b22a8a4306a276cdbc5: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-0/wal/wals/e3b7fe7e341a412d951d3bbd03a3ab28
I20260925 12:17:18.201608 2704 ts_tablet_manager.cc:1939] T e3b7fe7e341a412d951d3bbd03a3ab28 P 9268de0b968a4b22a8a4306a276cdbc5: Deleting consensus metadata
I20260925 12:17:18.201875 2103 catalog_manager.cc:4841] TS 9268de0b968a4b22a8a4306a276cdbc5 (127.0.98.1:35461): tablet e3b7fe7e341a412d951d3bbd03a3ab28 (table test-table [id=dcbe26dd11c94d00862018f37c2b06ef]) successfully deleted
I20260925 12:17:18.202400 2714 tablet_bootstrap.cc:492] T c6bfe727714e4aabb31fd13b45cb217f P 152e2c85bff341db9e4f6b7347af8034: Bootstrap starting.
I20260925 12:17:18.202406 2716 tablet_bootstrap.cc:492] T c6bfe727714e4aabb31fd13b45cb217f P 9268de0b968a4b22a8a4306a276cdbc5: Bootstrap starting.
I20260925 12:17:18.202664 2716 tablet_bootstrap.cc:654] T c6bfe727714e4aabb31fd13b45cb217f P 9268de0b968a4b22a8a4306a276cdbc5: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:18.202668 2394 tablet_service.cc:1467] Processing CreateTablet for tablet c6bfe727714e4aabb31fd13b45cb217f (DEFAULT_TABLE table=test-table [id=a9dd70701382449b94315db6ed98422b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:18.202705 2714 tablet_bootstrap.cc:654] T c6bfe727714e4aabb31fd13b45cb217f P 152e2c85bff341db9e4f6b7347af8034: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:18.202953 2705 ts_tablet_manager.cc:1918] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:17:18.203018 2705 log.cc:1196] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/e3b7fe7e341a412d951d3bbd03a3ab28
I20260925 12:17:18.203219 2705 ts_tablet_manager.cc:1939] T e3b7fe7e341a412d951d3bbd03a3ab28 P 04884f65bae6405f806102a42e6344a9: Deleting consensus metadata
I20260925 12:17:18.203253 2714 tablet_bootstrap.cc:492] T c6bfe727714e4aabb31fd13b45cb217f P 152e2c85bff341db9e4f6b7347af8034: No bootstrap required, opened a new log
I20260925 12:17:18.203293 2714 ts_tablet_manager.cc:1397] T c6bfe727714e4aabb31fd13b45cb217f P 152e2c85bff341db9e4f6b7347af8034: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:18.203358 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6bfe727714e4aabb31fd13b45cb217f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:18.203400 2714 raft_consensus.cc:348] T c6bfe727714e4aabb31fd13b45cb217f P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:18.203473 2714 raft_consensus.cc:374] T c6bfe727714e4aabb31fd13b45cb217f P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:18.203517 2714 raft_consensus.cc:729] T c6bfe727714e4aabb31fd13b45cb217f P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152e2c85bff341db9e4f6b7347af8034, State: Initialized, Role: FOLLOWER
I20260925 12:17:18.203559 2714 consensus_queue.cc:260] T c6bfe727714e4aabb31fd13b45cb217f P 152e2c85bff341db9e4f6b7347af8034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:18.203665 2714 ts_tablet_manager.cc:1428] T c6bfe727714e4aabb31fd13b45cb217f P 152e2c85bff341db9e4f6b7347af8034: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:18.204734 2715 tablet_bootstrap.cc:492] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9: Bootstrap starting.
I20260925 12:17:18.205127 2715 tablet_bootstrap.cc:654] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:18.205415 2716 tablet_bootstrap.cc:492] T c6bfe727714e4aabb31fd13b45cb217f P 9268de0b968a4b22a8a4306a276cdbc5: No bootstrap required, opened a new log
I20260925 12:17:18.205461 2716 ts_tablet_manager.cc:1397] T c6bfe727714e4aabb31fd13b45cb217f P 9268de0b968a4b22a8a4306a276cdbc5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 12:17:18.205525 2102 catalog_manager.cc:4841] TS 04884f65bae6405f806102a42e6344a9 (127.0.98.2:36637): tablet e3b7fe7e341a412d951d3bbd03a3ab28 (table test-table [id=dcbe26dd11c94d00862018f37c2b06ef]) successfully deleted
I20260925 12:17:18.205562 2716 raft_consensus.cc:348] T c6bfe727714e4aabb31fd13b45cb217f P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:18.205621 2716 raft_consensus.cc:374] T c6bfe727714e4aabb31fd13b45cb217f P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:18.205641 2716 raft_consensus.cc:729] T c6bfe727714e4aabb31fd13b45cb217f P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9268de0b968a4b22a8a4306a276cdbc5, State: Initialized, Role: FOLLOWER
I20260925 12:17:18.205681 2716 consensus_queue.cc:260] T c6bfe727714e4aabb31fd13b45cb217f P 9268de0b968a4b22a8a4306a276cdbc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:18.205760 2716 ts_tablet_manager.cc:1428] T c6bfe727714e4aabb31fd13b45cb217f P 9268de0b968a4b22a8a4306a276cdbc5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:18.205762 2715 tablet_bootstrap.cc:492] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9: No bootstrap required, opened a new log
I20260925 12:17:18.205799 2715 ts_tablet_manager.cc:1397] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:18.205937 2715 raft_consensus.cc:348] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:18.206007 2715 raft_consensus.cc:374] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:18.206022 2715 raft_consensus.cc:729] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04884f65bae6405f806102a42e6344a9, State: Initialized, Role: FOLLOWER
I20260925 12:17:18.206076 2715 consensus_queue.cc:260] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:18.206182 2715 ts_tablet_manager.cc:1428] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:18.404827 2720 raft_consensus.cc:482] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:17:18.404907 2720 raft_consensus.cc:504] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:18.405079 2720 leader_election.cc:288] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9268de0b968a4b22a8a4306a276cdbc5 (127.0.98.1:35461), 152e2c85bff341db9e4f6b7347af8034 (127.0.98.3:45487)
I20260925 12:17:18.405261 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6bfe727714e4aabb31fd13b45cb217f" candidate_uuid: "04884f65bae6405f806102a42e6344a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9268de0b968a4b22a8a4306a276cdbc5" is_pre_election: true
I20260925 12:17:18.405299 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6bfe727714e4aabb31fd13b45cb217f" candidate_uuid: "04884f65bae6405f806102a42e6344a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152e2c85bff341db9e4f6b7347af8034" is_pre_election: true
I20260925 12:17:18.405397 2274 raft_consensus.cc:2454] T c6bfe727714e4aabb31fd13b45cb217f P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04884f65bae6405f806102a42e6344a9 in term 0.
I20260925 12:17:18.405393 2553 raft_consensus.cc:2454] T c6bfe727714e4aabb31fd13b45cb217f P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04884f65bae6405f806102a42e6344a9 in term 0.
I20260925 12:17:18.405556 2347 leader_election.cc:302] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04884f65bae6405f806102a42e6344a9, 9268de0b968a4b22a8a4306a276cdbc5; no voters:
I20260925 12:17:18.405661 2720 raft_consensus.cc:2781] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:17:18.405707 2720 raft_consensus.cc:482] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:17:18.405735 2720 raft_consensus.cc:3037] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:18.406373 2720 raft_consensus.cc:504] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:18.406571 2720 leader_election.cc:288] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [CANDIDATE]: Term 1 election: Requested vote from peers 9268de0b968a4b22a8a4306a276cdbc5 (127.0.98.1:35461), 152e2c85bff341db9e4f6b7347af8034 (127.0.98.3:45487)
I20260925 12:17:18.406692 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6bfe727714e4aabb31fd13b45cb217f" candidate_uuid: "04884f65bae6405f806102a42e6344a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9268de0b968a4b22a8a4306a276cdbc5"
I20260925 12:17:18.406702 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6bfe727714e4aabb31fd13b45cb217f" candidate_uuid: "04884f65bae6405f806102a42e6344a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152e2c85bff341db9e4f6b7347af8034"
I20260925 12:17:18.406786 2274 raft_consensus.cc:3037] T c6bfe727714e4aabb31fd13b45cb217f P 9268de0b968a4b22a8a4306a276cdbc5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:18.406811 2553 raft_consensus.cc:3037] T c6bfe727714e4aabb31fd13b45cb217f P 152e2c85bff341db9e4f6b7347af8034 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:18.407509 2274 raft_consensus.cc:2454] T c6bfe727714e4aabb31fd13b45cb217f P 9268de0b968a4b22a8a4306a276cdbc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04884f65bae6405f806102a42e6344a9 in term 1.
I20260925 12:17:18.407503 2553 raft_consensus.cc:2454] T c6bfe727714e4aabb31fd13b45cb217f P 152e2c85bff341db9e4f6b7347af8034 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04884f65bae6405f806102a42e6344a9 in term 1.
I20260925 12:17:18.407701 2347 leader_election.cc:302] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04884f65bae6405f806102a42e6344a9, 9268de0b968a4b22a8a4306a276cdbc5; no voters:
I20260925 12:17:18.407819 2720 raft_consensus.cc:2781] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:18.407867 2720 raft_consensus.cc:686] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [term 1 LEADER]: Becoming Leader. State: Replica: 04884f65bae6405f806102a42e6344a9, State: Running, Role: LEADER
I20260925 12:17:18.407908 2720 consensus_queue.cc:237] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } }
I20260925 12:17:18.408457 2117 catalog_manager.cc:5495] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04884f65bae6405f806102a42e6344a9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "04884f65bae6405f806102a42e6344a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04884f65bae6405f806102a42e6344a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 } health_report { overall_health: UNKNOWN } } }
I20260925 12:17:18.433079 2274 raft_consensus.cc:1261] T c6bfe727714e4aabb31fd13b45cb217f P 9268de0b968a4b22a8a4306a276cdbc5 [term 1 FOLLOWER]: Refusing update from remote peer 04884f65bae6405f806102a42e6344a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:18.433087 2553 raft_consensus.cc:1261] T c6bfe727714e4aabb31fd13b45cb217f P 152e2c85bff341db9e4f6b7347af8034 [term 1 FOLLOWER]: Refusing update from remote peer 04884f65bae6405f806102a42e6344a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:18.433282 2720 consensus_queue.cc:1035] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9268de0b968a4b22a8a4306a276cdbc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:18.433383 2734 consensus_queue.cc:1035] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "152e2c85bff341db9e4f6b7347af8034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 12:17:18.437376 2686 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 12:17:19.956403 2116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:19.956521 2116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:19.957320 2116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 443dd8f6b0a341f6b9fbc72f794d6165: Sending DeleteTablet for 3 replicas of tablet c6bfe727714e4aabb31fd13b45cb217f
I20260925 12:17:19.957576 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet c6bfe727714e4aabb31fd13b45cb217f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56214
I20260925 12:17:19.957588 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet c6bfe727714e4aabb31fd13b45cb217f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37906
I20260925 12:17:19.957588 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet c6bfe727714e4aabb31fd13b45cb217f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44326
I20260925 12:17:19.957782 2766 tablet_replica.cc:330] T c6bfe727714e4aabb31fd13b45cb217f P 152e2c85bff341db9e4f6b7347af8034: stopping tablet replica
I20260925 12:17:19.957834 2765 tablet_replica.cc:330] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9: stopping tablet replica
I20260925 12:17:19.957844 2766 raft_consensus.cc:2229] T c6bfe727714e4aabb31fd13b45cb217f P 152e2c85bff341db9e4f6b7347af8034 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:19.957880 2766 raft_consensus.cc:2258] T c6bfe727714e4aabb31fd13b45cb217f P 152e2c85bff341db9e4f6b7347af8034 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:19.957942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 2189
I20260925 12:17:19.957887 2765 raft_consensus.cc:2229] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:17:19.958770 2765 raft_consensus.cc:2258] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:19.958990 2765 ts_tablet_manager.cc:1905] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:19.959882 2766 ts_tablet_manager.cc:1905] T c6bfe727714e4aabb31fd13b45cb217f P 152e2c85bff341db9e4f6b7347af8034: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:19.960850 2765 ts_tablet_manager.cc:1918] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 12:17:19.960914 2765 log.cc:1196] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/c6bfe727714e4aabb31fd13b45cb217f
I20260925 12:17:19.961164 2765 ts_tablet_manager.cc:1939] T c6bfe727714e4aabb31fd13b45cb217f P 04884f65bae6405f806102a42e6344a9: Deleting consensus metadata
I20260925 12:17:19.961566 2102 catalog_manager.cc:4841] TS 04884f65bae6405f806102a42e6344a9 (127.0.98.2:36637): tablet c6bfe727714e4aabb31fd13b45cb217f (table test-table [id=a9dd70701382449b94315db6ed98422b]) successfully deleted
I20260925 12:17:19.961659 2766 ts_tablet_manager.cc:1918] T c6bfe727714e4aabb31fd13b45cb217f P 152e2c85bff341db9e4f6b7347af8034: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 12:17:19.961709 2766 log.cc:1196] T c6bfe727714e4aabb31fd13b45cb217f P 152e2c85bff341db9e4f6b7347af8034: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/c6bfe727714e4aabb31fd13b45cb217f
I20260925 12:17:19.961915 2766 ts_tablet_manager.cc:1939] T c6bfe727714e4aabb31fd13b45cb217f P 152e2c85bff341db9e4f6b7347af8034: Deleting consensus metadata
I20260925 12:17:19.962271 2101 catalog_manager.cc:4841] TS 152e2c85bff341db9e4f6b7347af8034 (127.0.98.3:45487): tablet c6bfe727714e4aabb31fd13b45cb217f (table test-table [id=a9dd70701382449b94315db6ed98422b]) successfully deleted
W20260925 12:17:19.964272 2103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35461 (error 108)
I20260925 12:17:19.964507 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 2329
I20260925 12:17:19.965715 2769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:19.964614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35461 (local address 127.0.0.1:56224)
0925 12:17:19.965623 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:19.965669 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":986,"thread_start_us":59,"threads_started":1}
W20260925 12:17:19.965839 2103 catalog_manager.cc:4584] TS 9268de0b968a4b22a8a4306a276cdbc5 (127.0.98.1:35461): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6bfe727714e4aabb31fd13b45cb217f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35461: connect: Connection refused (error 111)
I20260925 12:17:19.970368 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 2469
I20260925 12:17:19.975224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 2085
2026-09-25T12:17:19Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5963 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:17:20 dist-test-slave-clmx krb5kdc[2772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:17:20 dist-test-slave-clmx krb5kdc[2772](info): set up 2 sockets
Sep 25 12:17:20 dist-test-slave-clmx krb5kdc[2772](info): commencing operation
krb5kdc: starting...
W20260925 12:17:20.955145 392 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:17:21 dist-test-slave-clmx krb5kdc[2772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:17:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:17:21.057507 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43697
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:21.132941 2785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:21.133172 2785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:21.133206 2785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:21.134306 2785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:21.134380 2785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:21.134403 2785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:21.134434 2785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:21.134462 2785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:17:21.135895 2785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:21.136155 2785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:21.136358 2785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:21.141139 2791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:21.141161 2785 server_base.cc:1034] running on GCE node
W20260925 12:17:21.141119 2790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:21.141129 2793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:21.141517 2785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:21.141795 2785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:21.142964 2785 hybrid_clock.cc:648] HybridClock initialized: now 1790338641142951 us; error 29 us; skew 500 ppm
Sep 25 12:17:21 dist-test-slave-clmx krb5kdc[2772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:21.145282 2785 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:17:21.145721 2785 webserver.cc:460] Webserver started at http://127.0.98.62:35359/ using document root <none> and password file <none>
I20260925 12:17:21.145926 2785 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:21.145987 2785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:21.146122 2785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:21.147075 2785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "6e37faa5027e4145890386f01e564140"
format_stamp: "Formatted at 2026-09-25 12:17:21 on dist-test-slave-clmx"
server_key: "23bdefd825fa9673289547a78e2fb2e8"
server_key_iv: "e406686c00dcb0ad04af6ab2deeb48db"
server_key_version: "encryptionkey@0"
I20260925 12:17:21.147408 2785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e37faa5027e4145890386f01e564140"
format_stamp: "Formatted at 2026-09-25 12:17:21 on dist-test-slave-clmx"
server_key: "23bdefd825fa9673289547a78e2fb2e8"
server_key_iv: "e406686c00dcb0ad04af6ab2deeb48db"
server_key_version: "encryptionkey@0"
I20260925 12:17:21.148715 2785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:17:21.149389 2800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:21.149550 2785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:21.149634 2785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "6e37faa5027e4145890386f01e564140"
format_stamp: "Formatted at 2026-09-25 12:17:21 on dist-test-slave-clmx"
server_key: "23bdefd825fa9673289547a78e2fb2e8"
server_key_iv: "e406686c00dcb0ad04af6ab2deeb48db"
server_key_version: "encryptionkey@0"
I20260925 12:17:21.149725 2785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:21.162700 2785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:21.164383 2785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:21.164527 2785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:21.168182 2785 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43697
I20260925 12:17:21.168211 2852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43697 every 8 connection(s)
I20260925 12:17:21.168642 2785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:21.169200 2853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:21.171133 2853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140: Bootstrap starting.
I20260925 12:17:21.171242 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 2785
I20260925 12:17:21.171339 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:17:21.171527 392 external_mini_cluster.cc:1428] Setting key 0997c5f20fd0bc5902bf6d8da40598c2
I20260925 12:17:21.171828 2853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:21.172183 2853 log.cc:824] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:21.172890 2853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140: No bootstrap required, opened a new log
I20260925 12:17:21.175105 2853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37faa5027e4145890386f01e564140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43697 } }
I20260925 12:17:21.175237 2853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:21.175259 2853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e37faa5027e4145890386f01e564140, State: Initialized, Role: FOLLOWER
I20260925 12:17:21.175361 2853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37faa5027e4145890386f01e564140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43697 } }
I20260925 12:17:21.175431 2853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:21.175451 2853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:21.175503 2853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:21.176100 2853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37faa5027e4145890386f01e564140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43697 } }
I20260925 12:17:21.176200 2853 leader_election.cc:302] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e37faa5027e4145890386f01e564140; no voters:
I20260925 12:17:21.176331 2853 leader_election.cc:288] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:17:21.176375 2858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:21.176518 2858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140 [term 1 LEADER]: Becoming Leader. State: Replica: 6e37faa5027e4145890386f01e564140, State: Running, Role: LEADER
I20260925 12:17:21.176550 2853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:21.176626 2858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37faa5027e4145890386f01e564140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43697 } }
I20260925 12:17:21.177027 2859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e37faa5027e4145890386f01e564140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37faa5027e4145890386f01e564140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43697 } } }
I20260925 12:17:21.177053 2860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e37faa5027e4145890386f01e564140. Latest consensus state: current_term: 1 leader_uuid: "6e37faa5027e4145890386f01e564140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37faa5027e4145890386f01e564140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43697 } } }
I20260925 12:17:21.177130 2859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:21.177135 2860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:21.177309 2862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:21.177775 2862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 12:17:21 dist-test-slave-clmx krb5kdc[2772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:21.179160 2862 catalog_manager.cc:1347] Generated new cluster ID: 4a88aad6a0f0469d977ba7453daf171c
I20260925 12:17:21.179210 2862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:17:21.182276 2856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:21.172001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35148 (local address 127.0.98.62:43697)
0925 12:17:21.172183 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:21.172191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:21.172223 (+ 32us) server_negotiation.cc:407] Connection header received
0925 12:17:21.172743 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:21.172751 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:21.172804 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:21.172945 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:21.173552 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:21.174982 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:21.175892 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:21.176492 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:21.176874 (+ 382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:21.179302 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:21.179305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:21.179305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:21.179314 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:21.180847 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:21.181210 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:21.181213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:21.181213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:21.181291 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:21.181505 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:21.181506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:21.181506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:21.181720 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:21.181888 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:21.181964 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 12:17:21.182104 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260925 12:17:21.205471 2862 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:17:21.206136 2862 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:21.211891 2862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e37faa5027e4145890386f01e564140: Generated new TSK 0
I20260925 12:17:21.212049 2862 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:17:21.240036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43697
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:21.314500 2879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:21.314725 2879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:21.314759 2879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:21.316053 2879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:21.316123 2879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:21.316169 2879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:17:21.317734 2879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:21.317960 2879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:21.318151 2879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:21.322479 2885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:21.322489 2884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:21.322527 2887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:21.322712 2879 server_base.cc:1034] running on GCE node
I20260925 12:17:21.322898 2879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:21.323153 2879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:21.324298 2879 hybrid_clock.cc:648] HybridClock initialized: now 1790338641324280 us; error 34 us; skew 500 ppm
Sep 25 12:17:21 dist-test-slave-clmx krb5kdc[2772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:21.333346 2879 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:17:21.333814 2879 webserver.cc:460] Webserver started at http://127.0.98.1:38951/ using document root <none> and password file <none>
I20260925 12:17:21.334021 2879 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:21.334081 2879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:21.334205 2879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:21.335158 2879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "e9f986193fd342e789735e51c75cad9d"
format_stamp: "Formatted at 2026-09-25 12:17:21 on dist-test-slave-clmx"
server_key: "18596496fc5c5d4cc4813d0e94a9040e"
server_key_iv: "c834b5a81d8e64b006f2916e01206264"
server_key_version: "encryptionkey@0"
I20260925 12:17:21.335523 2879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e9f986193fd342e789735e51c75cad9d"
format_stamp: "Formatted at 2026-09-25 12:17:21 on dist-test-slave-clmx"
server_key: "18596496fc5c5d4cc4813d0e94a9040e"
server_key_iv: "c834b5a81d8e64b006f2916e01206264"
server_key_version: "encryptionkey@0"
I20260925 12:17:21.336802 2879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:17:21.337508 2894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:21.337671 2879 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:17:21.337759 2879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "e9f986193fd342e789735e51c75cad9d"
format_stamp: "Formatted at 2026-09-25 12:17:21 on dist-test-slave-clmx"
server_key: "18596496fc5c5d4cc4813d0e94a9040e"
server_key_iv: "c834b5a81d8e64b006f2916e01206264"
server_key_version: "encryptionkey@0"
I20260925 12:17:21.337828 2879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:21.359472 2879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:21.361100 2879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:21.361269 2879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:21.361475 2879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:21.361749 2879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:21.361799 2879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:21.361837 2879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:21.361866 2879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:21.367273 2879 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40639
I20260925 12:17:21.367306 3007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40639 every 8 connection(s)
I20260925 12:17:21.367568 2879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 12:17:21 dist-test-slave-clmx krb5kdc[2772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:21.374191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 2879
I20260925 12:17:21.374284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:17:21.374444 392 external_mini_cluster.cc:1428] Setting key 32734ebcd6767766eeab1724be832e24
I20260925 12:17:21.375790 2856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:21.368287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40513 (local address 127.0.98.62:43697)
0925 12:17:21.368331 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:21.368333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:21.368491 (+ 158us) server_negotiation.cc:407] Connection header received
0925 12:17:21.369275 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:21.369276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:21.369285 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:21.369334 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:21.369829 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:21.370308 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:21.371176 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:21.371349 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:21.371394 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:21.373199 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:21.373201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:21.373201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:21.373207 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:21.374445 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:21.374843 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:21.374844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:21.374845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:21.374899 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:21.375178 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:21.375179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:21.375180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:21.375281 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:21.375345 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:21.375603 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 12:17:21.375709 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:17:21.375939 3010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:21.368238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43697 (local address 127.0.98.1:40513)
0925 12:17:21.368391 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:21.368426 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:21.369199 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:21.369384 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:21.369387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:21.369590 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:21.369746 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:21.369749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:21.370375 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:21.370375 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:21.371113 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:21.371114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:21.371421 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:21.371422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:21.371530 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:21.371923 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:21.371930 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:21.373124 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:21.374531 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:21.374536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:21.374538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:21.374787 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:21.374980 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:21.374981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:21.374981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:21.375100 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:21.375339 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:21.375344 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:21.375463 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 12:17:21.375584 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 12:17:21.375744 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260925 12:17:21.376690 3008 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43697
I20260925 12:17:21.376784 3008 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:21.376973 3008 heartbeater.cc:511] Master 127.0.98.62:43697 requested a full tablet report, sending...
I20260925 12:17:21.377471 2817 ts_manager.cc:194] Registered new tserver with Master: e9f986193fd342e789735e51c75cad9d (127.0.98.1:40639)
I20260925 12:17:21.378202 2817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40513
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:17:21.396034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43697
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:21.485816 3013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:21.486027 3013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:21.486063 3013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:21.487183 3013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:21.487241 3013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:21.487282 3013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:17:21.488735 3013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:21.488983 3013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:21.489178 3013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:21.493813 3019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:21.493846 3018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:21.493945 3013 server_base.cc:1034] running on GCE node
W20260925 12:17:21.493886 3021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:21.494247 3013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:21.494483 3013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:21.495662 3013 hybrid_clock.cc:648] HybridClock initialized: now 1790338641495646 us; error 26 us; skew 500 ppm
Sep 25 12:17:21 dist-test-slave-clmx krb5kdc[2772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:21.497864 3013 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:17:21.498226 3013 webserver.cc:460] Webserver started at http://127.0.98.2:37539/ using document root <none> and password file <none>
I20260925 12:17:21.498413 3013 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:21.498471 3013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:21.498611 3013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:21.499485 3013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "beeb4196041d41ff95f6f079386dd094"
format_stamp: "Formatted at 2026-09-25 12:17:21 on dist-test-slave-clmx"
server_key: "80dc565d22f0e1666a00d6d3288a266a"
server_key_iv: "d5cd9c94b06ee2a5c0c849b32085ca75"
server_key_version: "encryptionkey@0"
I20260925 12:17:21.499845 3013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "beeb4196041d41ff95f6f079386dd094"
format_stamp: "Formatted at 2026-09-25 12:17:21 on dist-test-slave-clmx"
server_key: "80dc565d22f0e1666a00d6d3288a266a"
server_key_iv: "d5cd9c94b06ee2a5c0c849b32085ca75"
server_key_version: "encryptionkey@0"
I20260925 12:17:21.500957 3013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:21.501592 3028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:21.501744 3013 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:21.501832 3013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "beeb4196041d41ff95f6f079386dd094"
format_stamp: "Formatted at 2026-09-25 12:17:21 on dist-test-slave-clmx"
server_key: "80dc565d22f0e1666a00d6d3288a266a"
server_key_iv: "d5cd9c94b06ee2a5c0c849b32085ca75"
server_key_version: "encryptionkey@0"
I20260925 12:17:21.501907 3013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:21.523784 3013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:21.525425 3013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:21.525579 3013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:21.525775 3013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:21.526055 3013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:21.526106 3013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:21.526145 3013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:21.526172 3013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:21.531512 3013 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43215
I20260925 12:17:21.531546 3141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43215 every 8 connection(s)
I20260925 12:17:21.531854 3013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:17:21 dist-test-slave-clmx krb5kdc[2772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:21.540050 2856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:21.532498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33477 (local address 127.0.98.62:43697)
0925 12:17:21.532539 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:21.532541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:21.532740 (+ 199us) server_negotiation.cc:407] Connection header received
0925 12:17:21.533602 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:21.533604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:21.533617 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:21.533665 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:21.534243 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:21.534733 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:21.535719 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:21.535917 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:21.535996 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:21.537781 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:21.537782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:21.537783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:21.537789 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:21.538804 (+ 1015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:21.539145 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:21.539146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:21.539147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:21.539199 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:21.539439 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:21.539440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:21.539440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:21.539545 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:21.539615 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:21.539878 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 12:17:21.539952 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:17:21.540199 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:21.532464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43697 (local address 127.0.98.2:33477)
0925 12:17:21.532627 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:21.532670 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:21.533489 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:21.533722 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:21.533726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:21.533955 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:21.534157 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:21.534161 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:21.534818 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:21.534820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:21.535649 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:21.535651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:21.536004 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:21.536005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:21.536124 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:21.536483 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:21.536490 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:21.537710 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:21.538884 (+ 1174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:21.538889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:21.538891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:21.539078 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:21.539264 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:21.539265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:21.539265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:21.539379 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:21.539606 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:21.539612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:21.539758 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 12:17:21.539858 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 12:17:21.539994 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260925 12:17:21.540632 3142 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43697
I20260925 12:17:21.540702 3142 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:21.540818 3142 heartbeater.cc:511] Master 127.0.98.62:43697 requested a full tablet report, sending...
I20260925 12:17:21.541148 2817 ts_manager.cc:194] Registered new tserver with Master: beeb4196041d41ff95f6f079386dd094 (127.0.98.2:43215)
I20260925 12:17:21.541190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 3013
I20260925 12:17:21.541270 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:17:21.541488 392 external_mini_cluster.cc:1428] Setting key aaf67c7708dacb4c402afcf902a00c40
I20260925 12:17:21.541653 2817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33477
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:17:21.562387 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43697
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:21.635782 3147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:21.635974 3147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:21.635996 3147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:21.637135 3147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:21.637172 3147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:21.637195 3147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:17:21.638608 3147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:21.638830 3147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:21.639032 3147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:21.643697 3152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:21.643719 3155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:21.643709 3153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:21.643941 3147 server_base.cc:1034] running on GCE node
I20260925 12:17:21.644187 3147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:21.644418 3147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:21.645583 3147 hybrid_clock.cc:648] HybridClock initialized: now 1790338641645568 us; error 41 us; skew 500 ppm
Sep 25 12:17:21 dist-test-slave-clmx krb5kdc[2772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:21.647845 3147 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:17:21.648229 3147 webserver.cc:460] Webserver started at http://127.0.98.3:45945/ using document root <none> and password file <none>
I20260925 12:17:21.648437 3147 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:21.648496 3147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:21.648636 3147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:21.649762 3147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "c4ba570a22274fdebf97fcf7fd7581d6"
format_stamp: "Formatted at 2026-09-25 12:17:21 on dist-test-slave-clmx"
server_key: "2fd1567aa2c16132932f3d4422b21b4f"
server_key_iv: "dc562854c9395ea1f132ee541f3dd980"
server_key_version: "encryptionkey@0"
I20260925 12:17:21.650112 3147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c4ba570a22274fdebf97fcf7fd7581d6"
format_stamp: "Formatted at 2026-09-25 12:17:21 on dist-test-slave-clmx"
server_key: "2fd1567aa2c16132932f3d4422b21b4f"
server_key_iv: "dc562854c9395ea1f132ee541f3dd980"
server_key_version: "encryptionkey@0"
I20260925 12:17:21.651328 3147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:17:21.652078 3162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:21.652237 3147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:21.652328 3147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "c4ba570a22274fdebf97fcf7fd7581d6"
format_stamp: "Formatted at 2026-09-25 12:17:21 on dist-test-slave-clmx"
server_key: "2fd1567aa2c16132932f3d4422b21b4f"
server_key_iv: "dc562854c9395ea1f132ee541f3dd980"
server_key_version: "encryptionkey@0"
I20260925 12:17:21.652403 3147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:21.674347 3147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:21.676033 3147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:21.676165 3147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:21.676366 3147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:21.676658 3147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:21.676697 3147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:21.676733 3147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:21.676762 3147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:21.682608 3147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43803
I20260925 12:17:21.682647 3275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43803 every 8 connection(s)
I20260925 12:17:21.682912 3147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:17:21.686419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 3147
I20260925 12:17:21.686532 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:17:21.686683 392 external_mini_cluster.cc:1428] Setting key 05fb7c5088eb4b18b905176e08983165
Sep 25 12:17:21 dist-test-slave-clmx krb5kdc[2772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:21.691195 2856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:21.683478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59929 (local address 127.0.98.62:43697)
0925 12:17:21.683545 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:21.683547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:21.683721 (+ 174us) server_negotiation.cc:407] Connection header received
0925 12:17:21.684598 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:21.684599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:21.684611 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:21.684659 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:21.685202 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:21.685690 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:21.686654 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:21.686881 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:21.686959 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:21.688774 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:21.688775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:21.688775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:21.688781 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:21.689988 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:21.690346 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:21.690348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:21.690348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:21.690397 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:21.690611 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:21.690612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:21.690612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:21.690732 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:21.690799 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:21.691038 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 12:17:21.691108 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 12:17:21.691357 3278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:21.683443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43697 (local address 127.0.98.3:59929)
0925 12:17:21.683590 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:21.683644 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:21.684492 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:21.684717 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:21.684721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:21.684944 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:21.685101 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:21.685105 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:21.685780 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:21.685781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:21.686543 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:21.686545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:21.686969 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:21.686970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:21.687075 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:21.687503 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:21.687510 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:21.688674 (+ 1164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:21.690077 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:21.690083 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:21.690084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:21.690270 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:21.690458 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:21.690459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:21.690459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:21.690554 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:21.690791 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:21.690796 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:21.690921 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 12:17:21.691019 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 12:17:21.691153 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260925 12:17:21.691910 3276 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43697
I20260925 12:17:21.691982 3276 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:21.692103 3276 heartbeater.cc:511] Master 127.0.98.62:43697 requested a full tablet report, sending...
I20260925 12:17:21.692396 2816 ts_manager.cc:194] Registered new tserver with Master: c4ba570a22274fdebf97fcf7fd7581d6 (127.0.98.3:43803)
I20260925 12:17:21.692760 2816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59929
I20260925 12:17:21.697850 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 12:17:21.707399 2856 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:21.706757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35162 (local address 127.0.98.62:43697)
0925 12:17:21.706804 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:21.706806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:21.706893 (+ 87us) server_negotiation.cc:407] Connection header received
0925 12:17:21.706930 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:21.706930 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:21.706938 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:21.706992 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:21.707361 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35162: BlockingRecv error: recv got EOF from 127.0.0.1:35162 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 12:17:21.708204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 2879
I20260925 12:17:21.712216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 3013
I20260925 12:17:21.716637 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 3147
I20260925 12:17:21.720396 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 2785
2026-09-25T12:17:21Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1744 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:17:21 dist-test-slave-clmx krb5kdc[3288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:17:21 dist-test-slave-clmx krb5kdc[3288](info): set up 2 sockets
Sep 25 12:17:21 dist-test-slave-clmx krb5kdc[3288](info): commencing operation
krb5kdc: starting...
W20260925 12:17:22.707150 392 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:17:22 dist-test-slave-clmx krb5kdc[3288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:17:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:17:22.809970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44733
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:22.883656 3301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:22.883867 3301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:22.883901 3301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:22.884999 3301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:22.885061 3301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:22.885083 3301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:22.885103 3301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:22.885129 3301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:17:22.886516 3301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:22.886762 3301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:22.886953 3301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:22.891299 3307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:22.891336 3309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:22.891333 3306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:22.891535 3301 server_base.cc:1034] running on GCE node
I20260925 12:17:22.891789 3301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:22.892066 3301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:22.893213 3301 hybrid_clock.cc:648] HybridClock initialized: now 1790338642893213 us; error 42 us; skew 500 ppm
Sep 25 12:17:22 dist-test-slave-clmx krb5kdc[3288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:22.895396 3301 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:17:22.895855 3301 webserver.cc:460] Webserver started at http://127.0.98.62:42117/ using document root <none> and password file <none>
I20260925 12:17:22.896049 3301 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:22.896104 3301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:22.896221 3301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:22.897094 3301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "4993c141c5f84a04ac0f4324dc1736bb"
format_stamp: "Formatted at 2026-09-25 12:17:22 on dist-test-slave-clmx"
server_key: "3e86d125640b1ff3d0341fc51ca72852"
server_key_iv: "0a11f99b9b7dbdf58551e4905020d7d3"
server_key_version: "encryptionkey@0"
I20260925 12:17:22.897424 3301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "4993c141c5f84a04ac0f4324dc1736bb"
format_stamp: "Formatted at 2026-09-25 12:17:22 on dist-test-slave-clmx"
server_key: "3e86d125640b1ff3d0341fc51ca72852"
server_key_iv: "0a11f99b9b7dbdf58551e4905020d7d3"
server_key_version: "encryptionkey@0"
I20260925 12:17:22.898552 3301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:17:22.899124 3316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:22.899276 3301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:22.899358 3301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "4993c141c5f84a04ac0f4324dc1736bb"
format_stamp: "Formatted at 2026-09-25 12:17:22 on dist-test-slave-clmx"
server_key: "3e86d125640b1ff3d0341fc51ca72852"
server_key_iv: "0a11f99b9b7dbdf58551e4905020d7d3"
server_key_version: "encryptionkey@0"
I20260925 12:17:22.899428 3301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:22.914300 3301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:22.915906 3301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:22.916045 3301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:22.919370 3301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44733
I20260925 12:17:22.919396 3368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44733 every 8 connection(s)
I20260925 12:17:22.919723 3301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:22.920431 3369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:22.922333 3369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb: Bootstrap starting.
I20260925 12:17:22.922873 3369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:22.923125 3369 log.cc:824] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:22.923482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 3301
I20260925 12:17:22.923566 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:17:22.923776 392 external_mini_cluster.cc:1428] Setting key 14acfb0f4e2135d9fa1e35ef368d0278
I20260925 12:17:22.923806 3369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb: No bootstrap required, opened a new log
I20260925 12:17:22.926383 3369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4993c141c5f84a04ac0f4324dc1736bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44733 } }
I20260925 12:17:22.926499 3369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:22.926520 3369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4993c141c5f84a04ac0f4324dc1736bb, State: Initialized, Role: FOLLOWER
I20260925 12:17:22.926630 3369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4993c141c5f84a04ac0f4324dc1736bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44733 } }
I20260925 12:17:22.926697 3369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:22.926730 3369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:22.926774 3369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:22.927327 3369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4993c141c5f84a04ac0f4324dc1736bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44733 } }
I20260925 12:17:22.927419 3369 leader_election.cc:302] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4993c141c5f84a04ac0f4324dc1736bb; no voters:
I20260925 12:17:22.927551 3369 leader_election.cc:288] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:17:22.927637 3374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:22.927773 3369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:22.927767 3374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb [term 1 LEADER]: Becoming Leader. State: Replica: 4993c141c5f84a04ac0f4324dc1736bb, State: Running, Role: LEADER
I20260925 12:17:22.928128 3374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4993c141c5f84a04ac0f4324dc1736bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44733 } }
I20260925 12:17:22.929448 3376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4993c141c5f84a04ac0f4324dc1736bb. Latest consensus state: current_term: 1 leader_uuid: "4993c141c5f84a04ac0f4324dc1736bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4993c141c5f84a04ac0f4324dc1736bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44733 } } }
I20260925 12:17:22.929461 3375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4993c141c5f84a04ac0f4324dc1736bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4993c141c5f84a04ac0f4324dc1736bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44733 } } }
I20260925 12:17:22.929544 3376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:22.929546 3375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb [sys.catalog]: This master's current role is: LEADER
W20260925 12:17:22.929831 3389 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:17:22.929883 3389 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:17:22.929940 3390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:22.930059 3390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 12:17:22 dist-test-slave-clmx krb5kdc[3288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:22.931512 3390 catalog_manager.cc:1347] Generated new cluster ID: 8aa4597901584f579e5112070f97bf65
I20260925 12:17:22.931588 3390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:17:22.934926 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:22.924259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38576 (local address 127.0.98.62:44733)
0925 12:17:22.924421 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:22.924428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:22.924466 (+ 38us) server_negotiation.cc:407] Connection header received
0925 12:17:22.925068 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:22.925074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:22.925118 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:22.925220 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:22.925924 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:22.926788 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:22.927742 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:22.928609 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:22.928746 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:22.931393 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:22.931396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:22.931396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:22.931406 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:22.933291 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:22.933636 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:22.933638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:22.933639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:22.933729 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:22.933880 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:22.933880 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:22.933880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:22.934153 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:22.934275 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:22.934650 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 12:17:22.934756 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260925 12:17:22.935667 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:17:22.946311 3390 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:17:22.946929 3390 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:22.949913 3390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4993c141c5f84a04ac0f4324dc1736bb: Generated new TSK 0
I20260925 12:17:22.950055 3390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:17:22.959318 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:22.952689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38580 (local address 127.0.98.62:44733)
0925 12:17:22.952738 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:22.952739 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:22.952856 (+ 117us) server_negotiation.cc:407] Connection header received
0925 12:17:22.952929 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:22.952930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:22.952939 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:22.952993 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:22.953308 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:22.953809 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:22.954532 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:22.954748 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:22.954822 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:22.958907 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:22.958908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:22.958908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:22.959078 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:22.959091 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:22.959097 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:22.959162 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:22.959221 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 12:17:22.959242 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 12:17:22.961249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 3301
2026-09-25T12:17:22Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1239 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:17:23 dist-test-slave-clmx krb5kdc[3402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:17:23 dist-test-slave-clmx krb5kdc[3402](info): set up 2 sockets
Sep 25 12:17:23 dist-test-slave-clmx krb5kdc[3402](info): commencing operation
krb5kdc: starting...
W20260925 12:17:23.947412 392 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:17:24 dist-test-slave-clmx krb5kdc[3402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:17:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:17:24.051056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39145
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:24.124352 3415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:24.124562 3415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:24.124600 3415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:24.126036 3415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:24.126124 3415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:24.126163 3415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:24.126191 3415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:24.126240 3415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:17:24.127734 3415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:24.127987 3415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:24.128199 3415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:24.132586 3420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:24.132587 3421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:24.132611 3423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:24.132858 3415 server_base.cc:1034] running on GCE node
I20260925 12:17:24.133070 3415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:24.133337 3415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:24.134474 3415 hybrid_clock.cc:648] HybridClock initialized: now 1790338644134460 us; error 31 us; skew 500 ppm
Sep 25 12:17:24 dist-test-slave-clmx krb5kdc[3402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:24.136890 3415 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:17:24.137272 3415 webserver.cc:460] Webserver started at http://127.0.98.62:41589/ using document root <none> and password file <none>
I20260925 12:17:24.137439 3415 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:24.137476 3415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:24.137555 3415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:24.138409 3415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "668bfe72f22b454880f904255ed66bf1"
format_stamp: "Formatted at 2026-09-25 12:17:24 on dist-test-slave-clmx"
server_key: "478aaaf346344d92345778ebddf9d406"
server_key_iv: "cdd72e533dafb32bf3f08c36fd8a6727"
server_key_version: "encryptionkey@0"
I20260925 12:17:24.138685 3415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "668bfe72f22b454880f904255ed66bf1"
format_stamp: "Formatted at 2026-09-25 12:17:24 on dist-test-slave-clmx"
server_key: "478aaaf346344d92345778ebddf9d406"
server_key_iv: "cdd72e533dafb32bf3f08c36fd8a6727"
server_key_version: "encryptionkey@0"
I20260925 12:17:24.139938 3415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:17:24.140584 3430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:24.140774 3415 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:17:24.140904 3415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "668bfe72f22b454880f904255ed66bf1"
format_stamp: "Formatted at 2026-09-25 12:17:24 on dist-test-slave-clmx"
server_key: "478aaaf346344d92345778ebddf9d406"
server_key_iv: "cdd72e533dafb32bf3f08c36fd8a6727"
server_key_version: "encryptionkey@0"
I20260925 12:17:24.140983 3415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:24.167987 3415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:24.170055 3415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:24.170194 3415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:24.173815 3415 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39145
I20260925 12:17:24.173846 3482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39145 every 8 connection(s)
I20260925 12:17:24.174144 3415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:24.174991 3483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:24.175130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 3415
I20260925 12:17:24.175218 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:17:24.175417 392 external_mini_cluster.cc:1428] Setting key 6da080d96c1e67b81e7d52c1f7d3fe2c
I20260925 12:17:24.177842 3483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1: Bootstrap starting.
I20260925 12:17:24.178444 3483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:24.178716 3483 log.cc:824] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:24.179283 3483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1: No bootstrap required, opened a new log
Sep 25 12:17:24 dist-test-slave-clmx krb5kdc[3402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:24.181577 3483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668bfe72f22b454880f904255ed66bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39145 } }
I20260925 12:17:24.181733 3483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:24.181777 3483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668bfe72f22b454880f904255ed66bf1, State: Initialized, Role: FOLLOWER
I20260925 12:17:24.181922 3483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668bfe72f22b454880f904255ed66bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39145 } }
I20260925 12:17:24.182022 3483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:24.182065 3483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:24.182114 3483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:24.182587 3483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668bfe72f22b454880f904255ed66bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39145 } }
I20260925 12:17:24.182685 3483 leader_election.cc:302] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 668bfe72f22b454880f904255ed66bf1; no voters:
I20260925 12:17:24.182849 3483 leader_election.cc:288] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:17:24.182905 3488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:24.183079 3488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1 [term 1 LEADER]: Becoming Leader. State: Replica: 668bfe72f22b454880f904255ed66bf1, State: Running, Role: LEADER
I20260925 12:17:24.183089 3483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:24.183172 3488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668bfe72f22b454880f904255ed66bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39145 } }
I20260925 12:17:24.183470 3488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 668bfe72f22b454880f904255ed66bf1. Latest consensus state: current_term: 1 leader_uuid: "668bfe72f22b454880f904255ed66bf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668bfe72f22b454880f904255ed66bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39145 } } }
I20260925 12:17:24.183477 3489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "668bfe72f22b454880f904255ed66bf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668bfe72f22b454880f904255ed66bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39145 } } }
I20260925 12:17:24.183593 3489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:24.183717 3488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:24.184116 3495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:24.184592 3495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:17:24.185710 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:24.175869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37252 (local address 127.0.98.62:39145)
0925 12:17:24.176011 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:24.176018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:24.176039 (+ 21us) server_negotiation.cc:407] Connection header received
0925 12:17:24.176413 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:24.176418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:24.176458 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:24.176555 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:24.177236 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:24.178039 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:24.178964 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:24.179381 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:24.179489 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:24.181769 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:24.181773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:24.181773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:24.181785 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:24.183749 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:24.184164 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:24.184182 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:24.184182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:24.184280 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:24.184487 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:24.184488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:24.184489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:24.184751 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:24.184887 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:24.185369 (+ 482us) server_negotiation.cc:299] Negotiation successful
0925 12:17:24.185518 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260925 12:17:24.186318 3495 catalog_manager.cc:1347] Generated new cluster ID: b2cb6e4badbc43e990408b9d36a1834a
I20260925 12:17:24.186364 3495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:17:24.198315 3495 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:17:24.199039 3495 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:24.204576 3495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 668bfe72f22b454880f904255ed66bf1: Generated new TSK 0
I20260925 12:17:24.204732 3495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:17:24.228075 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39145
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:24.302634 3509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:24.302824 3509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:24.302855 3509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:24.303855 3509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:24.303886 3509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:24.303911 3509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:17:24.305253 3509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:24.305409 3509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:24.305558 3509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:24.309903 3517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:24.309965 3515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:24.309993 3509 server_base.cc:1034] running on GCE node
W20260925 12:17:24.309908 3514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:24.310268 3509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:24.310494 3509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:24.311674 3509 hybrid_clock.cc:648] HybridClock initialized: now 1790338644311668 us; error 31 us; skew 500 ppm
Sep 25 12:17:24 dist-test-slave-clmx krb5kdc[3402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:24.313896 3509 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:17:24.314270 3509 webserver.cc:460] Webserver started at http://127.0.98.1:33663/ using document root <none> and password file <none>
I20260925 12:17:24.314487 3509 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:24.314565 3509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:24.314704 3509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:24.315883 3509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "a035ec4223d94fa9ae224f2fbad564d1"
format_stamp: "Formatted at 2026-09-25 12:17:24 on dist-test-slave-clmx"
server_key: "49d3315640306c935a5ff0cd85189f1b"
server_key_iv: "d6ae3070ab5a8a5da77f4a02b7420199"
server_key_version: "encryptionkey@0"
I20260925 12:17:24.316227 3509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a035ec4223d94fa9ae224f2fbad564d1"
format_stamp: "Formatted at 2026-09-25 12:17:24 on dist-test-slave-clmx"
server_key: "49d3315640306c935a5ff0cd85189f1b"
server_key_iv: "d6ae3070ab5a8a5da77f4a02b7420199"
server_key_version: "encryptionkey@0"
I20260925 12:17:24.317355 3509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:24.318030 3524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:24.318179 3509 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:24.318261 3509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "a035ec4223d94fa9ae224f2fbad564d1"
format_stamp: "Formatted at 2026-09-25 12:17:24 on dist-test-slave-clmx"
server_key: "49d3315640306c935a5ff0cd85189f1b"
server_key_iv: "d6ae3070ab5a8a5da77f4a02b7420199"
server_key_version: "encryptionkey@0"
I20260925 12:17:24.318332 3509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:24.329912 3509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:24.331470 3509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:24.331668 3509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:24.331866 3509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:24.332131 3509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:24.332180 3509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:24.332212 3509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:24.332238 3509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:24.337607 3509 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41005
I20260925 12:17:24.337669 3637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41005 every 8 connection(s)
I20260925 12:17:24.337942 3509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:17:24.341782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 3509
I20260925 12:17:24.341871 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:17:24.342080 392 external_mini_cluster.cc:1428] Setting key 63f91b7c6a1a46b97075dae7af32b531
Sep 25 12:17:24 dist-test-slave-clmx krb5kdc[3402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:24.346760 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:24.338581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43101 (local address 127.0.98.62:39145)
0925 12:17:24.338635 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:24.338637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:24.338805 (+ 168us) server_negotiation.cc:407] Connection header received
0925 12:17:24.339220 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:24.339221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:24.339232 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:24.339292 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:24.340069 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:24.340513 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:24.341409 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:24.341653 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:24.341731 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:24.343913 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:24.343915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:24.343916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:24.343925 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:24.345379 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:24.345811 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:24.345812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:24.345812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:24.345855 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:24.346148 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:24.346149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:24.346149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:24.346278 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:24.346334 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:24.346643 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 12:17:24.346706 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 12:17:24.347028 3640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:24.338562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39145 (local address 127.0.98.1:43101)
0925 12:17:24.338713 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:24.338739 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:24.339112 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:24.339354 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:24.339357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:24.339826 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:24.339979 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:24.339983 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:24.340606 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:24.340608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:24.341330 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:24.341332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:24.341735 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:24.341736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:24.341848 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:24.342332 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:24.342341 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:24.343819 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:24.345457 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:24.345463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:24.345465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:24.345748 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:24.345942 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:24.345943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:24.345943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:24.346076 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:24.346346 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:24.346352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:24.346504 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 12:17:24.346627 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 12:17:24.346773 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260925 12:17:24.347496 3638 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39145
I20260925 12:17:24.347585 3638 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:24.347793 3638 heartbeater.cc:511] Master 127.0.98.62:39145 requested a full tablet report, sending...
I20260925 12:17:24.348338 3447 ts_manager.cc:194] Registered new tserver with Master: a035ec4223d94fa9ae224f2fbad564d1 (127.0.98.1:41005)
I20260925 12:17:24.349236 3447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43101
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:17:24.364289 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39145
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:24.437764 3643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:24.437932 3643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:24.437951 3643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:24.438956 3643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:24.438992 3643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:24.439035 3643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:17:24.440339 3643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:24.440538 3643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:24.440732 3643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:24.445220 3651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:24.445284 3643 server_base.cc:1034] running on GCE node
W20260925 12:17:24.445230 3648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:24.445220 3649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:24.445604 3643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:24.445804 3643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:24.446941 3643 hybrid_clock.cc:648] HybridClock initialized: now 1790338644446928 us; error 26 us; skew 500 ppm
Sep 25 12:17:24 dist-test-slave-clmx krb5kdc[3402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:24.449244 3643 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:17:24.449561 3643 webserver.cc:460] Webserver started at http://127.0.98.2:40619/ using document root <none> and password file <none>
I20260925 12:17:24.449718 3643 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:24.449753 3643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:24.449831 3643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:24.450628 3643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "1c4a271a61a04da1a948c8b8474daeb1"
format_stamp: "Formatted at 2026-09-25 12:17:24 on dist-test-slave-clmx"
server_key: "9fa38dc85774ac7a430f2b2277c223d0"
server_key_iv: "cdf2c0afb0c3cb95e95947ac2b54c1a5"
server_key_version: "encryptionkey@0"
I20260925 12:17:24.450886 3643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1c4a271a61a04da1a948c8b8474daeb1"
format_stamp: "Formatted at 2026-09-25 12:17:24 on dist-test-slave-clmx"
server_key: "9fa38dc85774ac7a430f2b2277c223d0"
server_key_iv: "cdf2c0afb0c3cb95e95947ac2b54c1a5"
server_key_version: "encryptionkey@0"
I20260925 12:17:24.452054 3643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:17:24.452718 3658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:24.452879 3643 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:17:24.452967 3643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "1c4a271a61a04da1a948c8b8474daeb1"
format_stamp: "Formatted at 2026-09-25 12:17:24 on dist-test-slave-clmx"
server_key: "9fa38dc85774ac7a430f2b2277c223d0"
server_key_iv: "cdf2c0afb0c3cb95e95947ac2b54c1a5"
server_key_version: "encryptionkey@0"
I20260925 12:17:24.453016 3643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:24.474866 3643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:24.476825 3643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:24.476949 3643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:24.477092 3643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:24.477349 3643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:24.477381 3643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:24.477399 3643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:24.477449 3643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:24.482522 3643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36879
I20260925 12:17:24.482573 3771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36879 every 8 connection(s)
I20260925 12:17:24.482860 3643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:17:24.488122 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 3643
Sep 25 12:17:24 dist-test-slave-clmx krb5kdc[3402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:24.488216 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:17:24.488394 392 external_mini_cluster.cc:1428] Setting key b589a7e27d5e8650692501085de809fa
I20260925 12:17:24.491348 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:24.483499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60829 (local address 127.0.98.62:39145)
0925 12:17:24.483555 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:24.483556 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:24.483807 (+ 251us) server_negotiation.cc:407] Connection header received
0925 12:17:24.484249 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:24.484250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:24.484259 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:24.484307 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:24.485152 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:24.485621 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:24.486471 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:24.486660 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:24.486706 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:24.488845 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:24.488845 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:24.488845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:24.488850 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:24.490146 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:24.490563 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:24.490564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:24.490564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:24.490607 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:24.490804 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:24.490804 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:24.490805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:24.490886 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:24.490936 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:24.491235 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 12:17:24.491290 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 12:17:24.491608 3774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:24.483498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39145 (local address 127.0.98.2:60829)
0925 12:17:24.483688 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:24.483725 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:24.484173 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:24.484388 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:24.484392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:24.484915 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:24.485085 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:24.485089 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:24.485688 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:24.485689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:24.486401 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:24.486403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:24.486719 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:24.486720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:24.486806 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:24.487189 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:24.487196 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:24.488573 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:24.490213 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:24.490219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:24.490220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:24.490497 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:24.490646 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:24.490647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:24.490647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:24.490755 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:24.490940 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:24.490944 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:24.491092 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 12:17:24.491218 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 12:17:24.491373 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260925 12:17:24.492089 3772 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39145
I20260925 12:17:24.492180 3772 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:24.492363 3772 heartbeater.cc:511] Master 127.0.98.62:39145 requested a full tablet report, sending...
I20260925 12:17:24.492674 3447 ts_manager.cc:194] Registered new tserver with Master: 1c4a271a61a04da1a948c8b8474daeb1 (127.0.98.2:36879)
I20260925 12:17:24.493037 3447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60829
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:17:24.510210 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39145
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:24.585278 3777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:24.585474 3777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:24.585496 3777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:24.586550 3777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:24.586582 3777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:24.586607 3777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:17:24.587888 3777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:24.588047 3777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:24.588203 3777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:17:24.592425 3777 server_base.cc:1034] running on GCE node
W20260925 12:17:24.592286 3782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:24.592306 3785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:24.592308 3783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:24.592746 3777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:24.592985 3777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:24.594116 3777 hybrid_clock.cc:648] HybridClock initialized: now 1790338644594101 us; error 32 us; skew 500 ppm
Sep 25 12:17:24 dist-test-slave-clmx krb5kdc[3402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:24.596333 3777 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:17:24.596709 3777 webserver.cc:460] Webserver started at http://127.0.98.3:37189/ using document root <none> and password file <none>
I20260925 12:17:24.596872 3777 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:24.596908 3777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:24.596989 3777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:24.597777 3777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "abd3f8f134a747499fc0d58b298a027e"
format_stamp: "Formatted at 2026-09-25 12:17:24 on dist-test-slave-clmx"
server_key: "b386e9c7e82b133bd36f44d12d87c332"
server_key_iv: "a622ec6bb83767084fdea4aa5a6adfa9"
server_key_version: "encryptionkey@0"
I20260925 12:17:24.598029 3777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "abd3f8f134a747499fc0d58b298a027e"
format_stamp: "Formatted at 2026-09-25 12:17:24 on dist-test-slave-clmx"
server_key: "b386e9c7e82b133bd36f44d12d87c332"
server_key_iv: "a622ec6bb83767084fdea4aa5a6adfa9"
server_key_version: "encryptionkey@0"
I20260925 12:17:24.599080 3777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:17:24.599653 3792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:24.599807 3777 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:24.599888 3777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "abd3f8f134a747499fc0d58b298a027e"
format_stamp: "Formatted at 2026-09-25 12:17:24 on dist-test-slave-clmx"
server_key: "b386e9c7e82b133bd36f44d12d87c332"
server_key_iv: "a622ec6bb83767084fdea4aa5a6adfa9"
server_key_version: "encryptionkey@0"
I20260925 12:17:24.599960 3777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:24.612128 3777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:24.613698 3777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:24.613837 3777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:24.614023 3777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:24.614310 3777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:24.614357 3777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:24.614395 3777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:24.614423 3777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:24.619793 3777 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40529
I20260925 12:17:24.619833 3905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40529 every 8 connection(s)
I20260925 12:17:24.620100 3777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:17:24.624214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 3777
I20260925 12:17:24.624318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:17:24.624538 392 external_mini_cluster.cc:1428] Setting key 99acc3edc2013911f9456efb07ade918
Sep 25 12:17:24 dist-test-slave-clmx krb5kdc[3402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:24.628329 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:24.620682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59553 (local address 127.0.98.62:39145)
0925 12:17:24.620718 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:24.620721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:24.620869 (+ 148us) server_negotiation.cc:407] Connection header received
0925 12:17:24.621351 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:24.621353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:24.621364 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:24.621413 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:24.622264 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:24.622761 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:24.623729 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:24.623916 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:24.623993 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:24.626182 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:24.626183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:24.626183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:24.626189 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:24.627269 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:24.627593 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:24.627593 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:24.627594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:24.627684 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:24.627891 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:24.627891 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:24.627892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:24.627985 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:24.628035 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:24.628220 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 12:17:24.628273 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 12:17:24.628563 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:24.620651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39145 (local address 127.0.98.3:59553)
0925 12:17:24.620766 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:24.620801 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:24.621254 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:24.621469 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:24.621472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:24.622013 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:24.622176 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:24.622179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:24.622854 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:24.622855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:24.623609 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:24.623649 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:24.624001 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:24.624003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:24.624168 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:24.624626 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:24.624636 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:24.626109 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:24.627330 (+ 1221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:24.627334 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:24.627336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:24.627534 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:24.627736 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:24.627736 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:24.627736 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:24.627840 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:24.628035 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:24.628038 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:24.628134 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 12:17:24.628209 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 12:17:24.628311 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":57,"threads_started":1}
I20260925 12:17:24.628975 3906 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39145
I20260925 12:17:24.629056 3906 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:24.629176 3906 heartbeater.cc:511] Master 127.0.98.62:39145 requested a full tablet report, sending...
I20260925 12:17:24.629478 3447 ts_manager.cc:194] Registered new tserver with Master: abd3f8f134a747499fc0d58b298a027e (127.0.98.3:40529)
I20260925 12:17:24.629822 3447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59553
I20260925 12:17:24.636209 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 12:17:24 dist-test-slave-clmx krb5kdc[3402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338644, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 12:17:24 dist-test-slave-clmx krb5kdc[3402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338644, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:24.656014 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:24.650010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37258 (local address 127.0.98.62:39145)
0925 12:17:24.650061 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:24.650063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:24.650172 (+ 109us) server_negotiation.cc:407] Connection header received
0925 12:17:24.650215 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:24.650216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:24.650223 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:24.650268 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:24.650693 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:24.651067 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:24.651722 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:24.651891 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:24.651940 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:24.653937 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:24.653938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:24.653938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:24.653943 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:24.655014 (+ 1071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:24.655276 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:24.655277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:24.655277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:24.655320 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:24.655450 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:24.655450 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:24.655450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:24.655554 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:24.655661 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:24.655827 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 12:17:24.655896 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 12:17:24.656198 3447 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37258
I20260925 12:17:24.656548 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 3509
I20260925 12:17:24.660666 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 3643
I20260925 12:17:24.664683 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 3777
I20260925 12:17:24.668650 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 3415
2026-09-25T12:17:24Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1709 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:17:24 dist-test-slave-clmx krb5kdc[3918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:17:24 dist-test-slave-clmx krb5kdc[3918](info): set up 2 sockets
Sep 25 12:17:24 dist-test-slave-clmx krb5kdc[3918](info): commencing operation
krb5kdc: starting...
W20260925 12:17:25.647389 392 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:17:25 dist-test-slave-clmx krb5kdc[3918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:17:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:17:25.748104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45175
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:25.823154 3931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:25.823333 3931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:25.823354 3931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:25.824445 3931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:25.824479 3931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:25.824493 3931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:25.824505 3931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:25.824517 3931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:17:25.825816 3931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:25.825977 3931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:25.826136 3931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:25.830440 3937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:25.830469 3939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:25.830456 3936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:25.830693 3931 server_base.cc:1034] running on GCE node
I20260925 12:17:25.830936 3931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:25.831200 3931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:25.832337 3931 hybrid_clock.cc:648] HybridClock initialized: now 1790338645832329 us; error 32 us; skew 500 ppm
Sep 25 12:17:25 dist-test-slave-clmx krb5kdc[3918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:25.834491 3931 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:17:25.834892 3931 webserver.cc:460] Webserver started at http://127.0.98.62:45859/ using document root <none> and password file <none>
I20260925 12:17:25.835085 3931 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:25.835140 3931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:25.835256 3931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:25.836228 3931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "641c9083e2e24b47b9a4ac21f070bdf6"
format_stamp: "Formatted at 2026-09-25 12:17:25 on dist-test-slave-clmx"
server_key: "61847f4241d1f049474242e4f906ad6a"
server_key_iv: "6024888539767bec8f91fe50d4c7b458"
server_key_version: "encryptionkey@0"
I20260925 12:17:25.836560 3931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "641c9083e2e24b47b9a4ac21f070bdf6"
format_stamp: "Formatted at 2026-09-25 12:17:25 on dist-test-slave-clmx"
server_key: "61847f4241d1f049474242e4f906ad6a"
server_key_iv: "6024888539767bec8f91fe50d4c7b458"
server_key_version: "encryptionkey@0"
I20260925 12:17:25.837745 3931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:17:25.838370 3946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:25.838518 3931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:25.838605 3931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "641c9083e2e24b47b9a4ac21f070bdf6"
format_stamp: "Formatted at 2026-09-25 12:17:25 on dist-test-slave-clmx"
server_key: "61847f4241d1f049474242e4f906ad6a"
server_key_iv: "6024888539767bec8f91fe50d4c7b458"
server_key_version: "encryptionkey@0"
I20260925 12:17:25.838675 3931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:25.850111 3931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:25.851596 3931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:25.851794 3931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:25.855268 3931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45175
I20260925 12:17:25.855295 3998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45175 every 8 connection(s)
I20260925 12:17:25.855584 3931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:25.856324 3999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:25.858196 3999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6: Bootstrap starting.
I20260925 12:17:25.858623 3999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:25.858824 3999 log.cc:824] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:25.859350 3999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6: No bootstrap required, opened a new log
I20260925 12:17:25.860713 3999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641c9083e2e24b47b9a4ac21f070bdf6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } }
I20260925 12:17:25.860826 3999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:25.860843 3999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 641c9083e2e24b47b9a4ac21f070bdf6, State: Initialized, Role: FOLLOWER
I20260925 12:17:25.860958 3999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641c9083e2e24b47b9a4ac21f070bdf6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } }
I20260925 12:17:25.861025 3999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:25.861044 3999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:25.861068 3999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:25.861459 3999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641c9083e2e24b47b9a4ac21f070bdf6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } }
I20260925 12:17:25.861528 3999 leader_election.cc:302] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 641c9083e2e24b47b9a4ac21f070bdf6; no voters:
I20260925 12:17:25.861673 3999 leader_election.cc:288] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:17:25.861701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 3931
I20260925 12:17:25.861725 4002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:25.861814 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:17:25.861927 3999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:25.862037 392 external_mini_cluster.cc:1428] Setting key 4bae55686bfbda636d6868ced32c8740
I20260925 12:17:25.862100 4002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6 [term 1 LEADER]: Becoming Leader. State: Replica: 641c9083e2e24b47b9a4ac21f070bdf6, State: Running, Role: LEADER
I20260925 12:17:25.862179 4002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641c9083e2e24b47b9a4ac21f070bdf6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } }
I20260925 12:17:25.862527 4002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 641c9083e2e24b47b9a4ac21f070bdf6. Latest consensus state: current_term: 1 leader_uuid: "641c9083e2e24b47b9a4ac21f070bdf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641c9083e2e24b47b9a4ac21f070bdf6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } } }
I20260925 12:17:25.862671 4002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:25.862759 4003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "641c9083e2e24b47b9a4ac21f070bdf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641c9083e2e24b47b9a4ac21f070bdf6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } } }
I20260925 12:17:25.862835 4003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:25.862990 4013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:25.863667 4013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:17:25.865206 4013 catalog_manager.cc:1347] Generated new cluster ID: c5f3c17657f94e46846e154a8773a8ed
I20260925 12:17:25.865257 4013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:17:25 dist-test-slave-clmx krb5kdc[3918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:25.872854 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:25.862622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46446 (local address 127.0.98.62:45175)
0925 12:17:25.862932 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:25.862953 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:25.862988 (+ 35us) server_negotiation.cc:407] Connection header received
0925 12:17:25.863115 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:25.863121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:25.863174 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:25.863293 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:25.864036 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:25.865317 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:25.866301 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:25.866845 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:25.866932 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:25.869545 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:25.869548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:25.869549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:25.869559 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:25.871325 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:25.871715 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:25.871718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:25.871718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:25.871828 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:25.872038 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:25.872039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:25.872039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:25.872297 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:25.872430 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:25.872533 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 12:17:25.872652 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":78,"threads_started":1}
I20260925 12:17:25.887817 4013 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:17:25.888239 4013 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:25.892795 4013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 641c9083e2e24b47b9a4ac21f070bdf6: Generated new TSK 0
I20260925 12:17:25.892984 4013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:17:25.918115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45175
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:25.991806 4025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:25.991986 4025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:25.992009 4025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:25.993130 4025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:25.993168 4025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:25.993235 4025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:17:25.994666 4025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:25.994856 4025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:25.995092 4025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:25.999426 4031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:25.999444 4033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:25.999433 4030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:25.999640 4025 server_base.cc:1034] running on GCE node
I20260925 12:17:25.999833 4025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:26.000036 4025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:26.001165 4025 hybrid_clock.cc:648] HybridClock initialized: now 1790338646001147 us; error 34 us; skew 500 ppm
Sep 25 12:17:25 dist-test-slave-clmx krb5kdc[3918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:26.003381 4025 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:17:26.003829 4025 webserver.cc:460] Webserver started at http://127.0.98.1:39111/ using document root <none> and password file <none>
I20260925 12:17:26.004029 4025 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:26.004117 4025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:26.004242 4025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:26.005165 4025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "b155745d8d5b44db8daa60c80412199c"
format_stamp: "Formatted at 2026-09-25 12:17:26 on dist-test-slave-clmx"
server_key: "632c9dc7fda20932c28fd78dbbbfb263"
server_key_iv: "d1cc3c19bf5d9f7c3d105a253b15e356"
server_key_version: "encryptionkey@0"
I20260925 12:17:26.005515 4025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b155745d8d5b44db8daa60c80412199c"
format_stamp: "Formatted at 2026-09-25 12:17:26 on dist-test-slave-clmx"
server_key: "632c9dc7fda20932c28fd78dbbbfb263"
server_key_iv: "d1cc3c19bf5d9f7c3d105a253b15e356"
server_key_version: "encryptionkey@0"
I20260925 12:17:26.006705 4025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:17:26.007326 4040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:26.007485 4025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:26.007573 4025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "b155745d8d5b44db8daa60c80412199c"
format_stamp: "Formatted at 2026-09-25 12:17:26 on dist-test-slave-clmx"
server_key: "632c9dc7fda20932c28fd78dbbbfb263"
server_key_iv: "d1cc3c19bf5d9f7c3d105a253b15e356"
server_key_version: "encryptionkey@0"
I20260925 12:17:26.007659 4025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:26.024253 4025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:26.025831 4025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:26.025974 4025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:26.026166 4025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:26.026429 4025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:26.026476 4025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:26.026506 4025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:26.026532 4025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:26.031720 4025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35049
I20260925 12:17:26.031757 4153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35049 every 8 connection(s)
I20260925 12:17:26.032056 4025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 12:17:26 dist-test-slave-clmx krb5kdc[3918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:26.039819 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:26.032657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32793 (local address 127.0.98.62:45175)
0925 12:17:26.032717 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:26.032719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:26.032885 (+ 166us) server_negotiation.cc:407] Connection header received
0925 12:17:26.033736 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:26.033738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:26.033750 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:26.033788 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:26.034311 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:26.034770 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:26.035641 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:26.035824 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:26.035870 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:26.037674 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:26.037676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:26.037676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:26.037683 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:26.038790 (+ 1107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:26.039109 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:26.039109 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:26.039109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:26.039153 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:26.039335 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:26.039336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:26.039336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:26.039423 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:26.039466 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:26.039704 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 12:17:26.039762 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 12:17:26.039940 4156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:26.032646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45175 (local address 127.0.98.1:32793)
0925 12:17:26.032782 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:26.032811 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:26.033649 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:26.033853 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:26.033856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:26.034088 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:26.034247 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:26.034250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:26.034835 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:26.034836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:26.035556 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:26.035557 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:26.035885 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:26.035886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:26.035973 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:26.036324 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:26.036330 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:26.037579 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:26.038852 (+ 1273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:26.038857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:26.038858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:26.039052 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:26.039214 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:26.039214 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:26.039214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:26.039293 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:26.039470 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:26.039473 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:26.039562 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 12:17:26.039687 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 12:17:26.039787 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260925 12:17:26.040367 4154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45175
I20260925 12:17:26.040465 4154 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:26.040614 4154 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
I20260925 12:17:26.041086 3963 ts_manager.cc:194] Registered new tserver with Master: b155745d8d5b44db8daa60c80412199c (127.0.98.1:35049)
I20260925 12:17:26.041610 3963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32793
I20260925 12:17:26.041883 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 4025
I20260925 12:17:26.041990 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:17:26.042171 392 external_mini_cluster.cc:1428] Setting key 4906b7edd7882318e8a5fda791959849
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:17:26.062233 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45175
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:26.135185 4159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:26.135367 4159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:26.135401 4159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:26.136474 4159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:26.136530 4159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:26.136570 4159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:17:26.137938 4159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:26.138175 4159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:26.138386 4159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:26.143153 4167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:26.143213 4164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:26.143270 4159 server_base.cc:1034] running on GCE node
W20260925 12:17:26.143191 4165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:26.143481 4159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:26.143705 4159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:26.144850 4159 hybrid_clock.cc:648] HybridClock initialized: now 1790338646144831 us; error 29 us; skew 500 ppm
Sep 25 12:17:26 dist-test-slave-clmx krb5kdc[3918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:26.147284 4159 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:17:26.147716 4159 webserver.cc:460] Webserver started at http://127.0.98.2:46249/ using document root <none> and password file <none>
I20260925 12:17:26.147935 4159 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:26.147997 4159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:26.148123 4159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:26.149065 4159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "a077b0373b274b9ab866f07953389ad4"
format_stamp: "Formatted at 2026-09-25 12:17:26 on dist-test-slave-clmx"
server_key: "fb0a5ee5d2af5b18672fab2e4d9afab4"
server_key_iv: "25da963880996d8d83e72ae1fc0e94c3"
server_key_version: "encryptionkey@0"
I20260925 12:17:26.149433 4159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a077b0373b274b9ab866f07953389ad4"
format_stamp: "Formatted at 2026-09-25 12:17:26 on dist-test-slave-clmx"
server_key: "fb0a5ee5d2af5b18672fab2e4d9afab4"
server_key_iv: "25da963880996d8d83e72ae1fc0e94c3"
server_key_version: "encryptionkey@0"
I20260925 12:17:26.150640 4159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:17:26.151309 4174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:26.151463 4159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:26.151551 4159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "a077b0373b274b9ab866f07953389ad4"
format_stamp: "Formatted at 2026-09-25 12:17:26 on dist-test-slave-clmx"
server_key: "fb0a5ee5d2af5b18672fab2e4d9afab4"
server_key_iv: "25da963880996d8d83e72ae1fc0e94c3"
server_key_version: "encryptionkey@0"
I20260925 12:17:26.151648 4159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:26.158524 4159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:26.160063 4159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:26.160205 4159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:26.160375 4159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:26.160648 4159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:26.160696 4159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:26.160732 4159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:26.160759 4159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:26.166126 4159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41507
I20260925 12:17:26.166189 4287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41507 every 8 connection(s)
I20260925 12:17:26.166473 4159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:17:26 dist-test-slave-clmx krb5kdc[3918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:26.174173 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:26.167104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44947 (local address 127.0.98.62:45175)
0925 12:17:26.167146 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:26.167148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:26.167319 (+ 171us) server_negotiation.cc:407] Connection header received
0925 12:17:26.168210 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:26.168211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:26.168221 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:26.168283 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:26.168766 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:26.169243 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:26.170130 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:26.170300 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:26.170346 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:26.172060 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:26.172061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:26.172061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:26.172067 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:26.173197 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:26.173518 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:26.173519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:26.173519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:26.173562 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:26.173740 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:26.173741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:26.173741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:26.173825 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:26.173869 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:26.174056 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 12:17:26.174116 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:17:26.174326 4290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:26.167074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45175 (local address 127.0.98.2:44947)
0925 12:17:26.167211 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:26.167241 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:26.168127 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:26.168327 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:26.168330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:26.168547 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:26.168704 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:26.168708 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:26.169320 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:26.169320 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:26.170065 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:26.170067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:26.170360 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:26.170361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:26.170456 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:26.170790 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:26.170797 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:26.171981 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:26.173257 (+ 1276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:26.173262 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:26.173264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:26.173461 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:26.173611 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:26.173611 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:26.173611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:26.173695 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:26.173871 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:26.173874 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:26.173966 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 12:17:26.174037 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 12:17:26.174133 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":86,"threads_started":1}
I20260925 12:17:26.174739 4288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45175
I20260925 12:17:26.174821 4288 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:26.174966 4288 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
I20260925 12:17:26.175307 3963 ts_manager.cc:194] Registered new tserver with Master: a077b0373b274b9ab866f07953389ad4 (127.0.98.2:41507)
I20260925 12:17:26.175688 3963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44947
I20260925 12:17:26.176182 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 4159
I20260925 12:17:26.176263 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:17:26.176440 392 external_mini_cluster.cc:1428] Setting key d12074cff88571324d05810467b0d09e
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:17:26.197804 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45175
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:26.273048 4293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:26.273211 4293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:26.273231 4293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:26.274211 4293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:26.274240 4293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:26.274262 4293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:17:26.276155 4293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:26.276314 4293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:26.276468 4293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:26.280766 4301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:26.280860 4299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:26.280771 4298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:26.280869 4293 server_base.cc:1034] running on GCE node
I20260925 12:17:26.281183 4293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:26.281356 4293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:26.282503 4293 hybrid_clock.cc:648] HybridClock initialized: now 1790338646282489 us; error 38 us; skew 500 ppm
Sep 25 12:17:26 dist-test-slave-clmx krb5kdc[3918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:26.284682 4293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:17:26.285104 4293 webserver.cc:460] Webserver started at http://127.0.98.3:39083/ using document root <none> and password file <none>
I20260925 12:17:26.285310 4293 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:26.285369 4293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:26.285499 4293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:26.286439 4293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "2703eac7706244b6804e4b0b2c21b597"
format_stamp: "Formatted at 2026-09-25 12:17:26 on dist-test-slave-clmx"
server_key: "c1d06fe2e5f53bf87f0529f10b5bc081"
server_key_iv: "d7e916f9966205c1be5967eab27040ed"
server_key_version: "encryptionkey@0"
I20260925 12:17:26.286753 4293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2703eac7706244b6804e4b0b2c21b597"
format_stamp: "Formatted at 2026-09-25 12:17:26 on dist-test-slave-clmx"
server_key: "c1d06fe2e5f53bf87f0529f10b5bc081"
server_key_iv: "d7e916f9966205c1be5967eab27040ed"
server_key_version: "encryptionkey@0"
I20260925 12:17:26.287976 4293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 12:17:26.288691 4308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:26.288859 4293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:26.288954 4293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "2703eac7706244b6804e4b0b2c21b597"
format_stamp: "Formatted at 2026-09-25 12:17:26 on dist-test-slave-clmx"
server_key: "c1d06fe2e5f53bf87f0529f10b5bc081"
server_key_iv: "d7e916f9966205c1be5967eab27040ed"
server_key_version: "encryptionkey@0"
I20260925 12:17:26.289024 4293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:26.307433 4293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:26.309055 4293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:26.309201 4293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:26.309417 4293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:26.309701 4293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:26.309762 4293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:26.309793 4293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:26.309818 4293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:26.315119 4293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42935
I20260925 12:17:26.315171 4421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42935 every 8 connection(s)
I20260925 12:17:26.315423 4293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 12:17:26 dist-test-slave-clmx krb5kdc[3918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:26.321521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 4293
I20260925 12:17:26.321616 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:17:26.321830 392 external_mini_cluster.cc:1428] Setting key ebfa45c8cfdf11d2552f03db2171eaab
I20260925 12:17:26.323715 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:26.316044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57739 (local address 127.0.98.62:45175)
0925 12:17:26.316100 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:26.316102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:26.316258 (+ 156us) server_negotiation.cc:407] Connection header received
0925 12:17:26.317069 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:26.317070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:26.317079 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:26.317131 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:26.317664 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:26.318150 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:26.319026 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:26.319194 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:26.319240 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:26.321070 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:26.321071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:26.321072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:26.321078 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:26.322570 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:26.322978 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:26.322979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:26.322979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:26.323028 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:26.323198 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:26.323198 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:26.323199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:26.323287 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:26.323342 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:26.323535 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 12:17:26.323590 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 12:17:26.323819 4424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:26.316010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45175 (local address 127.0.98.3:57739)
0925 12:17:26.316153 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:26.316188 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:26.316989 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:26.317189 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:26.317192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:26.317417 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:26.317582 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:26.317586 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:26.318218 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:26.318218 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:26.318960 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:26.318962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:26.319254 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:26.319255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:26.319339 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:26.319770 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:26.319777 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:26.320962 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:26.322639 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:26.322656 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:26.322658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:26.322905 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:26.323077 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:26.323078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:26.323078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:26.323155 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:26.323334 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:26.323336 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:26.323440 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 12:17:26.323512 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 12:17:26.323610 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260925 12:17:26.324236 4422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45175
I20260925 12:17:26.324328 4422 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:26.324487 4422 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
I20260925 12:17:26.324803 3963 ts_manager.cc:194] Registered new tserver with Master: 2703eac7706244b6804e4b0b2c21b597 (127.0.98.3:42935)
I20260925 12:17:26.325150 3963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57739
I20260925 12:17:26.333536 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 12:17:26 dist-test-slave-clmx krb5kdc[3918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 12:17:26 dist-test-slave-clmx krb5kdc[3918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:17:26.354226 4429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:26.347623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.0.98.1:35049)
0925 12:17:26.347790 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:26.347794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:26.347803 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:17:26.347830 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:26.347832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:26.347845 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:26.347905 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:26.348369 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:26.349023 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:26.349678 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:26.349947 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:26.349994 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:26.351980 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:26.351986 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:26.351986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:26.351992 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:26.353260 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:26.353569 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:26.353571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:26.353571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:26.353621 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:26.353778 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:26.353779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:26.353779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:26.353926 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:26.354046 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:26.354100 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 12:17:26.354157 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
Sep 25 12:17:26 dist-test-slave-clmx krb5kdc[3918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:26.360811 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:26.355239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46462 (local address 127.0.98.62:45175)
0925 12:17:26.355271 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:26.355273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:26.355334 (+ 61us) server_negotiation.cc:407] Connection header received
0925 12:17:26.355423 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:26.355424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:26.355433 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:26.355496 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:26.355995 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:26.356404 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:26.356997 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:26.357168 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:26.357263 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:26.358964 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:26.358965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:26.358965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:26.358970 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:26.360027 (+ 1057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:26.360281 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:26.360281 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:26.360282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:26.360322 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:26.360450 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:26.360450 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:26.360450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:26.360549 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:26.360610 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:26.360676 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 12:17:26.360725 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
W20260925 12:17:26.361021 3963 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46462
I20260925 12:17:26.361304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 4025
I20260925 12:17:26.366829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 4159
I20260925 12:17:26.370702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 4293
I20260925 12:17:26.374528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 3931
2026-09-25T12:17:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1705 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:17:26 dist-test-slave-clmx krb5kdc[4434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:17:26 dist-test-slave-clmx krb5kdc[4434](info): set up 2 sockets
Sep 25 12:17:26 dist-test-slave-clmx krb5kdc[4434](info): commencing operation
krb5kdc: starting...
W20260925 12:17:27.367661 392 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:17:27 dist-test-slave-clmx krb5kdc[4434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:17:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:17:27.474591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39129
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:27.549822 4447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:27.549996 4447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:27.550019 4447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:27.551070 4447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:27.551111 4447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:27.551127 4447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:27.551189 4447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:27.551209 4447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:17:27.552816 4447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:27.553053 4447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:27.553308 4447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:27.557966 4452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:27.557986 4453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:27.558069 4447 server_base.cc:1034] running on GCE node
W20260925 12:17:27.557986 4455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:27.558328 4447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:27.558566 4447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:27.559733 4447 hybrid_clock.cc:648] HybridClock initialized: now 1790338647559726 us; error 26 us; skew 500 ppm
Sep 25 12:17:27 dist-test-slave-clmx krb5kdc[4434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:27.561780 4447 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:17:27.562161 4447 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:27.562228 4447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:27.562363 4447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:27.563319 4447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "14c153a337dc48d5b82896ef92f526f8"
format_stamp: "Formatted at 2026-09-25 12:17:27 on dist-test-slave-clmx"
server_key: "5c36e306acd2fb73cb17979ed2c52021"
server_key_iv: "560954ca0540624b82805672c8bd519a"
server_key_version: "encryptionkey@0"
I20260925 12:17:27.563696 4447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "14c153a337dc48d5b82896ef92f526f8"
format_stamp: "Formatted at 2026-09-25 12:17:27 on dist-test-slave-clmx"
server_key: "5c36e306acd2fb73cb17979ed2c52021"
server_key_iv: "560954ca0540624b82805672c8bd519a"
server_key_version: "encryptionkey@0"
I20260925 12:17:27.564952 4447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:27.565912 4461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:27.566097 4447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:27.566188 4447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "14c153a337dc48d5b82896ef92f526f8"
format_stamp: "Formatted at 2026-09-25 12:17:27 on dist-test-slave-clmx"
server_key: "5c36e306acd2fb73cb17979ed2c52021"
server_key_iv: "560954ca0540624b82805672c8bd519a"
server_key_version: "encryptionkey@0"
I20260925 12:17:27.566269 4447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:27.587890 4447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:27.589387 4447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:27.589551 4447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:27.593127 4447 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39129
I20260925 12:17:27.593160 4513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39129 every 8 connection(s)
I20260925 12:17:27.593389 4447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:27.594201 4514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:27.596094 4514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8: Bootstrap starting.
I20260925 12:17:27.596593 4514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:27.596861 4514 log.cc:824] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:27.597400 4514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8: No bootstrap required, opened a new log
I20260925 12:17:27.598385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 4447
I20260925 12:17:27.598462 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:17:27.598620 392 external_mini_cluster.cc:1428] Setting key 761cc92c86f8d159e13dbdb4f8ef0a0b
I20260925 12:17:27.598984 4514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c153a337dc48d5b82896ef92f526f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39129 } }
I20260925 12:17:27.599136 4514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:27.599205 4514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c153a337dc48d5b82896ef92f526f8, State: Initialized, Role: FOLLOWER
I20260925 12:17:27.599361 4514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c153a337dc48d5b82896ef92f526f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39129 } }
I20260925 12:17:27.599426 4514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:27.599453 4514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:27.599493 4514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:27.600101 4514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c153a337dc48d5b82896ef92f526f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39129 } }
I20260925 12:17:27.600230 4514 leader_election.cc:302] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14c153a337dc48d5b82896ef92f526f8; no voters:
I20260925 12:17:27.600381 4514 leader_election.cc:288] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:17:27.600443 4519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:27.600598 4519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8 [term 1 LEADER]: Becoming Leader. State: Replica: 14c153a337dc48d5b82896ef92f526f8, State: Running, Role: LEADER
I20260925 12:17:27.600613 4514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:27.600740 4519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c153a337dc48d5b82896ef92f526f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39129 } }
I20260925 12:17:27.601303 4521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14c153a337dc48d5b82896ef92f526f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c153a337dc48d5b82896ef92f526f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39129 } } }
I20260925 12:17:27.601397 4521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:27.601727 4520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14c153a337dc48d5b82896ef92f526f8. Latest consensus state: current_term: 1 leader_uuid: "14c153a337dc48d5b82896ef92f526f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c153a337dc48d5b82896ef92f526f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39129 } } }
I20260925 12:17:27.601850 4520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:27.601914 4527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:27.602450 4527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:17:27.604218 4527 catalog_manager.cc:1347] Generated new cluster ID: a62604ebba224c47b9ce1dff3daca792
I20260925 12:17:27.604267 4527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:17:27 dist-test-slave-clmx krb5kdc[4434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:27.609814 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:27.599104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37374 (local address 127.0.98.62:39129)
0925 12:17:27.599273 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:27.599282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:27.599310 (+ 28us) server_negotiation.cc:407] Connection header received
0925 12:17:27.599429 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:27.599436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:27.599485 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:27.599608 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:27.600193 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:27.602111 (+ 1918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:27.603137 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:27.603789 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:27.603885 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:27.606405 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:27.606408 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:27.606409 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:27.606419 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:27.608243 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:27.608616 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:27.608620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:27.608621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:27.608765 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:27.608965 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:27.608966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:27.608966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:27.609266 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:27.609436 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:27.609504 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 12:17:27.609629 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260925 12:17:27.615545 4527 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:17:27.616019 4527 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:27.621871 4527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8: Generated new TSK 0
I20260925 12:17:27.622044 4527 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:17:27.656173 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39129
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:27.738837 4540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:27.739001 4540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:27.739022 4540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:27.740149 4540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:27.740185 4540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:27.740226 4540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:17:27.741621 4540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:27.741850 4540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:27.742060 4540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:17:27.746539 4540 server_base.cc:1034] running on GCE node
W20260925 12:17:27.746500 4548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:27.746521 4545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:27.746501 4546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:27.746971 4540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:27.747217 4540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:27.748356 4540 hybrid_clock.cc:648] HybridClock initialized: now 1790338647748355 us; error 30 us; skew 500 ppm
Sep 25 12:17:27 dist-test-slave-clmx krb5kdc[4434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:27.750540 4540 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:17:27.750813 4540 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:27.750880 4540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:27.751009 4540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:27.751924 4540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "b595d6b257f840e4936e0745ec98f0c4"
format_stamp: "Formatted at 2026-09-25 12:17:27 on dist-test-slave-clmx"
server_key: "048968aea36d8a8dfad6d95258104d56"
server_key_iv: "95ea1d68af2c5f0087bcb25a8244111f"
server_key_version: "encryptionkey@0"
I20260925 12:17:27.752274 4540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b595d6b257f840e4936e0745ec98f0c4"
format_stamp: "Formatted at 2026-09-25 12:17:27 on dist-test-slave-clmx"
server_key: "048968aea36d8a8dfad6d95258104d56"
server_key_iv: "95ea1d68af2c5f0087bcb25a8244111f"
server_key_version: "encryptionkey@0"
I20260925 12:17:27.753479 4540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:17:27.754101 4554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:27.754246 4540 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:27.754340 4540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "b595d6b257f840e4936e0745ec98f0c4"
format_stamp: "Formatted at 2026-09-25 12:17:27 on dist-test-slave-clmx"
server_key: "048968aea36d8a8dfad6d95258104d56"
server_key_iv: "95ea1d68af2c5f0087bcb25a8244111f"
server_key_version: "encryptionkey@0"
I20260925 12:17:27.754427 4540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:27.761854 4540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:27.763438 4540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:27.763579 4540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:27.763804 4540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:27.764096 4540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:27.764153 4540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:27.764182 4540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:27.764209 4540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:27.769438 4540 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38619
I20260925 12:17:27.769495 4667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38619 every 8 connection(s)
I20260925 12:17:27.769709 4540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:17:27.770283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 4540
I20260925 12:17:27.770375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:17:27.770550 392 external_mini_cluster.cc:1428] Setting key 2ea342848947a0a7d0fcf378723a677c
Sep 25 12:17:27 dist-test-slave-clmx krb5kdc[4434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 12:17:27.779721 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:27.770916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47829 (local address 127.0.98.62:39129)
0925 12:17:27.770970 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:27.770972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:27.771126 (+ 154us) server_negotiation.cc:407] Connection header received
0925 12:17:27.771725 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:27.771726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:27.771736 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:27.771775 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:27.772976 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:27.773469 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:27.774536 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:27.774744 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:27.774825 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:27.776860 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:27.776862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:27.776862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:27.776870 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:27.778373 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:27.778774 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:27.778775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:27.778776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:27.778835 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:27.779098 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:27.779099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:27.779099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:27.779206 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:27.779279 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:27.779543 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 12:17:27.779638 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 12:17:27.779884 4670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:27.770898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39129 (local address 127.0.98.1:47829)
0925 12:17:27.771031 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:27.771065 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:27.771586 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:27.771848 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:27.771852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:27.772258 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:27.772873 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:27.772878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:27.773571 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:27.773572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:27.774427 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:27.774429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:27.774834 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:27.774835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:27.774965 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:27.775413 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:27.775422 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:27.776767 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:27.778482 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:27.778489 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:27.778490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:27.778697 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:27.778918 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:27.778919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:27.778919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:27.779043 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:27.779282 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:27.779288 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:27.779414 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 12:17:27.779510 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 12:17:27.779670 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 12:17:27.780395 4668 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39129
I20260925 12:17:27.780479 4668 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:27.780611 4668 heartbeater.cc:511] Master 127.0.98.62:39129 requested a full tablet report, sending...
I20260925 12:17:27.781085 4478 ts_manager.cc:194] Registered new tserver with Master: b595d6b257f840e4936e0745ec98f0c4 (127.0.98.1:38619)
I20260925 12:17:27.781818 4478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:17:27.792373 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39129
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:27.866640 4673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:27.866802 4673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:27.866822 4673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:27.867871 4673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:27.867902 4673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:27.867969 4673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:17:27.869374 4673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:27.869531 4673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:27.869673 4673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:27.874100 4679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:27.874082 4678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:27.874084 4681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:27.874348 4673 server_base.cc:1034] running on GCE node
I20260925 12:17:27.874552 4673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:27.874776 4673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:27.875931 4673 hybrid_clock.cc:648] HybridClock initialized: now 1790338647875912 us; error 29 us; skew 500 ppm
Sep 25 12:17:27 dist-test-slave-clmx krb5kdc[4434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:27.878250 4673 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:17:27.878594 4673 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:27.878655 4673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:27.878800 4673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:27.879774 4673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "c2856cc055204b689dc07b1c8dcafda3"
format_stamp: "Formatted at 2026-09-25 12:17:27 on dist-test-slave-clmx"
server_key: "1f3b120a3840888a46133617d668b498"
server_key_iv: "e5836c0008f34b60f20d4ccff01f503c"
server_key_version: "encryptionkey@0"
I20260925 12:17:27.880143 4673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c2856cc055204b689dc07b1c8dcafda3"
format_stamp: "Formatted at 2026-09-25 12:17:27 on dist-test-slave-clmx"
server_key: "1f3b120a3840888a46133617d668b498"
server_key_iv: "e5836c0008f34b60f20d4ccff01f503c"
server_key_version: "encryptionkey@0"
I20260925 12:17:27.881270 4673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:17:27.881899 4687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:27.882082 4673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:17:27.882174 4673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "c2856cc055204b689dc07b1c8dcafda3"
format_stamp: "Formatted at 2026-09-25 12:17:27 on dist-test-slave-clmx"
server_key: "1f3b120a3840888a46133617d668b498"
server_key_iv: "e5836c0008f34b60f20d4ccff01f503c"
server_key_version: "encryptionkey@0"
I20260925 12:17:27.882251 4673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:27.898824 4673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:27.901043 4673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:27.901201 4673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:27.901392 4673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:27.901697 4673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:27.901755 4673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:27.901793 4673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:27.901820 4673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:27.907447 4673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44017
I20260925 12:17:27.907490 4800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44017 every 8 connection(s)
I20260925 12:17:27.907724 4673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:17:27 dist-test-slave-clmx krb5kdc[4434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:27.915807 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:27.908322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56035 (local address 127.0.98.62:39129)
0925 12:17:27.908366 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:27.908368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:27.908538 (+ 170us) server_negotiation.cc:407] Connection header received
0925 12:17:27.908926 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:27.908927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:27.908936 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:27.908993 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:27.909753 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:27.910212 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:27.911082 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:27.911249 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:27.911295 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:27.913514 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:27.913515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:27.913516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:27.913522 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:27.914753 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:27.915076 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:27.915077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:27.915077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:27.915121 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:27.915305 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:27.915305 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:27.915305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:27.915388 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:27.915441 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:27.915693 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 12:17:27.915750 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:17:27.915931 4803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:27.908278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39129 (local address 127.0.98.2:56035)
0925 12:17:27.908416 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:27.908454 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:27.908841 (+ 387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:27.909039 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:27.909041 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:27.909256 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:27.909679 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:27.909684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:27.910284 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:27.910284 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:27.911012 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:27.911014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:27.911313 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:27.911314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:27.911401 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:27.911935 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:27.911943 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:27.913391 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:27.914819 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:27.914824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:27.914826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:27.915013 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:27.915174 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:27.915174 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:27.915175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:27.915256 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:27.915446 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:27.915449 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:27.915539 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 12:17:27.915609 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 12:17:27.915761 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260925 12:17:27.916379 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 4673
I20260925 12:17:27.916412 4801 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39129
I20260925 12:17:27.916493 4801 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:27.916495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:17:27.916630 4801 heartbeater.cc:511] Master 127.0.98.62:39129 requested a full tablet report, sending...
I20260925 12:17:27.916684 392 external_mini_cluster.cc:1428] Setting key 35113820126aa2a06c391c3dfc429eb2
I20260925 12:17:27.917014 4478 ts_manager.cc:194] Registered new tserver with Master: c2856cc055204b689dc07b1c8dcafda3 (127.0.98.2:44017)
I20260925 12:17:27.917450 4478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56035
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:17:27.938051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39129
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:28.012770 4806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:28.012981 4806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:28.013018 4806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:28.014147 4806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:28.014209 4806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:28.014254 4806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:17:28.015722 4806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:28.015985 4806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:28.016182 4806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:28.020577 4812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:28.020574 4811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:28.020740 4806 server_base.cc:1034] running on GCE node
W20260925 12:17:28.020629 4814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:28.020998 4806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:28.021225 4806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:28.022359 4806 hybrid_clock.cc:648] HybridClock initialized: now 1790338648022347 us; error 27 us; skew 500 ppm
Sep 25 12:17:28 dist-test-slave-clmx krb5kdc[4434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:28.029098 4806 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:17:28.029407 4806 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:28.029470 4806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:28.029596 4806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:28.030467 4806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "69302a8c35954e09a702fe35f042889c"
format_stamp: "Formatted at 2026-09-25 12:17:28 on dist-test-slave-clmx"
server_key: "51a694280405f3284a6eef0e280e1054"
server_key_iv: "7425a4cc2ae61f8edf963a9061db1f7e"
server_key_version: "encryptionkey@0"
I20260925 12:17:28.030786 4806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "69302a8c35954e09a702fe35f042889c"
format_stamp: "Formatted at 2026-09-25 12:17:28 on dist-test-slave-clmx"
server_key: "51a694280405f3284a6eef0e280e1054"
server_key_iv: "7425a4cc2ae61f8edf963a9061db1f7e"
server_key_version: "encryptionkey@0"
I20260925 12:17:28.032004 4806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:17:28.032670 4820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:28.032827 4806 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:28.032929 4806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "69302a8c35954e09a702fe35f042889c"
format_stamp: "Formatted at 2026-09-25 12:17:28 on dist-test-slave-clmx"
server_key: "51a694280405f3284a6eef0e280e1054"
server_key_iv: "7425a4cc2ae61f8edf963a9061db1f7e"
server_key_version: "encryptionkey@0"
I20260925 12:17:28.033006 4806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:28.057438 4806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:28.059446 4806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:28.059589 4806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:28.059827 4806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:28.060102 4806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:28.060153 4806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:28.060189 4806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:28.060216 4806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:28.065989 4806 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42041
I20260925 12:17:28.066015 4933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42041 every 8 connection(s)
I20260925 12:17:28.066252 4806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 12:17:28 dist-test-slave-clmx krb5kdc[4434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:28.071801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 4806
I20260925 12:17:28.071907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:17:28.072114 392 external_mini_cluster.cc:1428] Setting key 7b8cbe022e2fd9026044c52402243a7e
I20260925 12:17:28.074831 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.066873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36963 (local address 127.0.98.62:39129)
0925 12:17:28.066917 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:28.066919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:28.067068 (+ 149us) server_negotiation.cc:407] Connection header received
0925 12:17:28.067472 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:28.067473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:28.067481 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:28.067530 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:28.068376 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.068897 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.069849 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.070025 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.070076 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:28.072025 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:28.072027 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:28.072027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:28.072034 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:28.073546 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:28.073937 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:28.073938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:28.073939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:28.074000 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:28.074237 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:28.074237 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:28.074238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:28.074346 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:28.074420 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:28.074666 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 12:17:28.074749 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 12:17:28.074990 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.066833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39129 (local address 127.0.98.3:36963)
0925 12:17:28.066973 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:28.067007 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:28.067385 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:28.067572 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:28.067575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:28.067845 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:28.068300 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:28.068304 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.068987 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.068988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:28.069754 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:28.069756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.070090 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.070091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:28.070179 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:28.070628 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:28.070635 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:28.071874 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:28.073640 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:28.073646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:28.073648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:28.073844 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:28.074062 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:28.074063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:28.074063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:28.074166 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:28.074415 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:28.074421 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:28.074546 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 12:17:28.074645 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 12:17:28.074784 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":81,"threads_started":1}
I20260925 12:17:28.075351 4934 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39129
I20260925 12:17:28.075431 4934 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:28.075520 4934 heartbeater.cc:511] Master 127.0.98.62:39129 requested a full tablet report, sending...
I20260925 12:17:28.075817 4478 ts_manager.cc:194] Registered new tserver with Master: 69302a8c35954e09a702fe35f042889c (127.0.98.3:42041)
I20260925 12:17:28.076159 4478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36963
I20260925 12:17:28.083751 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:17:28.090102 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.084995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37378 (local address 127.0.98.62:39129)
0925 12:17:28.085075 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:28.085077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:28.085180 (+ 103us) server_negotiation.cc:407] Connection header received
0925 12:17:28.085214 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:28.085214 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:28.085222 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:28.085262 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:28.085761 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.086228 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.086896 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.087055 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.087098 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:28.088188 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:28.088188 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:28.088189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:28.088193 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:28.089251 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:28.089533 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:28.089534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:28.089534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:28.089576 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:28.089713 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:28.089714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:28.089714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:28.089815 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:28.089879 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:28.089973 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 12:17:28.090019 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 12:17:28.091581 4478 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:17:28.093099 4478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:17:28.098593 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.094990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44017 (local address 127.0.0.1:46908)
0925 12:17:28.095270 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:28.095307 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:28.095367 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:28.095533 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:28.095535 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:28.095537 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:28.095592 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:28.095593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.096553 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.096554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:28.097574 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:28.097576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.098309 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.098309 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:28.098374 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:28.098385 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 12:17:28.098451 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 12:17:28.098510 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":53,"threads_started":1}
I20260925 12:17:28.098749 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.095162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44316 (local address 127.0.98.1:38619)
0925 12:17:28.095312 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:28.095316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:28.095444 (+ 128us) server_negotiation.cc:407] Connection header received
0925 12:17:28.095484 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:28.095486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:28.095499 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:28.095560 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:28.095741 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.096752 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.097784 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.098401 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.098497 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:28.098546 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:28.098641 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 12:17:28.098677 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260925 12:17:28.098848 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.095003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38619 (local address 127.0.0.1:44316)
0925 12:17:28.095317 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:28.095327 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:28.095381 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:28.095656 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:28.095658 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:28.095659 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:28.095692 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:28.095693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.096840 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.096841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:28.097709 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:28.097711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.098509 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.098510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:28.098581 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:28.098584 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:28.098628 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 12:17:28.098774 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":57,"threads_started":1}
I20260925 12:17:28.098882 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.095104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46908 (local address 127.0.98.2:44017)
0925 12:17:28.095389 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:28.095394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:28.095403 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:17:28.095441 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:28.095443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:28.095457 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:28.095532 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:28.095678 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.096457 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.097646 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.098229 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.098297 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:28.098344 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:28.098765 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 12:17:28.098808 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":78,"threads_started":1}
I20260925 12:17:28.099443 4735 tablet_service.cc:1467] Processing CreateTablet for tablet 9e991a287c5e45cbb277af30e6c44c5d (DEFAULT_TABLE table=test-table [id=76d63fa799964301bc2718c94267ccb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:28.099578 4602 tablet_service.cc:1467] Processing CreateTablet for tablet 9e991a287c5e45cbb277af30e6c44c5d (DEFAULT_TABLE table=test-table [id=76d63fa799964301bc2718c94267ccb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:28.099794 4735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e991a287c5e45cbb277af30e6c44c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:28.099927 4602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e991a287c5e45cbb277af30e6c44c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:28.099985 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.095165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42041 (local address 127.0.0.1:47766)
0925 12:17:28.095362 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:28.095370 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:28.095406 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:28.095685 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:28.095686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:28.095688 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:28.095730 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:28.095731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.097783 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.097783 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:28.098512 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:28.098514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.099756 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.099757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:28.099814 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:28.099817 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:28.099864 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 12:17:28.099910 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
I20260925 12:17:28.100242 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.095245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47766 (local address 127.0.98.3:42041)
0925 12:17:28.095484 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:28.095487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:28.095494 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:17:28.095522 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:28.095523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:28.095534 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:28.095596 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:28.095790 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.096749 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.098586 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.099591 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.099677 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:28.099723 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:28.100121 (+ 398us) server_negotiation.cc:299] Negotiation successful
0925 12:17:28.100158 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":58,"threads_started":1}
I20260925 12:17:28.100749 4868 tablet_service.cc:1467] Processing CreateTablet for tablet 9e991a287c5e45cbb277af30e6c44c5d (DEFAULT_TABLE table=test-table [id=76d63fa799964301bc2718c94267ccb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:28.101025 4868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e991a287c5e45cbb277af30e6c44c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:28.102662 4948 tablet_bootstrap.cc:492] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4: Bootstrap starting.
I20260925 12:17:28.103220 4949 tablet_bootstrap.cc:492] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3: Bootstrap starting.
I20260925 12:17:28.103284 4948 tablet_bootstrap.cc:654] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:28.103790 4949 tablet_bootstrap.cc:654] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:28.103758 4948 log.cc:824] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:28.104377 4949 log.cc:824] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:28.104562 4948 tablet_bootstrap.cc:492] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4: No bootstrap required, opened a new log
I20260925 12:17:28.104640 4948 ts_tablet_manager.cc:1397] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 12:17:28.105018 4949 tablet_bootstrap.cc:492] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3: No bootstrap required, opened a new log
I20260925 12:17:28.105067 4949 ts_tablet_manager.cc:1397] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:17:28.106144 4948 raft_consensus.cc:348] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2856cc055204b689dc07b1c8dcafda3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "b595d6b257f840e4936e0745ec98f0c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38619 } } peers { permanent_uuid: "69302a8c35954e09a702fe35f042889c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 12:17:28.106295 4948 raft_consensus.cc:374] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:28.106338 4948 raft_consensus.cc:729] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b595d6b257f840e4936e0745ec98f0c4, State: Initialized, Role: FOLLOWER
I20260925 12:17:28.106451 4948 consensus_queue.cc:260] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2856cc055204b689dc07b1c8dcafda3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "b595d6b257f840e4936e0745ec98f0c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38619 } } peers { permanent_uuid: "69302a8c35954e09a702fe35f042889c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 12:17:28.106602 4949 raft_consensus.cc:348] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2856cc055204b689dc07b1c8dcafda3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "b595d6b257f840e4936e0745ec98f0c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38619 } } peers { permanent_uuid: "69302a8c35954e09a702fe35f042889c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 12:17:28.106714 4948 ts_tablet_manager.cc:1428] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 12:17:28.106739 4949 raft_consensus.cc:374] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:28.106781 4949 raft_consensus.cc:729] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2856cc055204b689dc07b1c8dcafda3, State: Initialized, Role: FOLLOWER
I20260925 12:17:28.106735 4668 heartbeater.cc:503] Master 127.0.98.62:39129 was elected leader, sending a full tablet report...
I20260925 12:17:28.104269 4951 tablet_bootstrap.cc:492] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c: Bootstrap starting.
I20260925 12:17:28.106895 4949 consensus_queue.cc:260] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2856cc055204b689dc07b1c8dcafda3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "b595d6b257f840e4936e0745ec98f0c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38619 } } peers { permanent_uuid: "69302a8c35954e09a702fe35f042889c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 12:17:28.107127 4949 ts_tablet_manager.cc:1428] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:17:28.107244 4801 heartbeater.cc:503] Master 127.0.98.62:39129 was elected leader, sending a full tablet report...
I20260925 12:17:28.107363 4951 tablet_bootstrap.cc:654] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:28.107981 4951 log.cc:824] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:28.108505 4951 tablet_bootstrap.cc:492] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c: No bootstrap required, opened a new log
I20260925 12:17:28.108573 4951 ts_tablet_manager.cc:1397] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260925 12:17:28.110114 4951 raft_consensus.cc:348] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2856cc055204b689dc07b1c8dcafda3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "b595d6b257f840e4936e0745ec98f0c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38619 } } peers { permanent_uuid: "69302a8c35954e09a702fe35f042889c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 12:17:28.110244 4951 raft_consensus.cc:374] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:28.110284 4951 raft_consensus.cc:729] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69302a8c35954e09a702fe35f042889c, State: Initialized, Role: FOLLOWER
I20260925 12:17:28.110402 4951 consensus_queue.cc:260] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2856cc055204b689dc07b1c8dcafda3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "b595d6b257f840e4936e0745ec98f0c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38619 } } peers { permanent_uuid: "69302a8c35954e09a702fe35f042889c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 12:17:28.110639 4951 ts_tablet_manager.cc:1428] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 12:17:28.110669 4934 heartbeater.cc:503] Master 127.0.98.62:39129 was elected leader, sending a full tablet report...
W20260925 12:17:28.158237 4802 tablet.cc:2369] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 12:17:28.270347 4669 tablet.cc:2369] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 12:17:28.316741 4935 tablet.cc:2369] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:17:28.374871 4953 raft_consensus.cc:482] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:17:28.375012 4953 raft_consensus.cc:504] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2856cc055204b689dc07b1c8dcafda3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "b595d6b257f840e4936e0745ec98f0c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38619 } } peers { permanent_uuid: "69302a8c35954e09a702fe35f042889c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 12:17:28.375363 4953 leader_election.cc:288] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2856cc055204b689dc07b1c8dcafda3 (127.0.98.2:44017), 69302a8c35954e09a702fe35f042889c (127.0.98.3:42041)
I20260925 12:17:28.377974 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.375501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44017 (local address 127.0.98.1:50901)
0925 12:17:28.375651 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:28.375675 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:28.375737 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:28.375868 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:28.375869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:28.375871 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:28.375939 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:28.375941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.376493 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.376494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:28.377174 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:28.377176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.377705 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.377708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:28.377783 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:28.377787 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:17:28.377839 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 12:17:28.377877 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260925 12:17:28.377981 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.375630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50901 (local address 127.0.98.2:44017)
0925 12:17:28.375688 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:28.375690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:28.375744 (+ 54us) server_negotiation.cc:407] Connection header received
0925 12:17:28.375789 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:28.375789 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:28.375800 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:28.375864 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:28.376004 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.376419 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.377260 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.377581 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.377654 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:28.377664 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:28.377860 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 12:17:28.377903 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 12:17:28.378072 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.375598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39987 (local address 127.0.98.3:42041)
0925 12:17:28.375666 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:28.375668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:28.375724 (+ 56us) server_negotiation.cc:407] Connection header received
0925 12:17:28.375811 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:28.375811 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:28.375824 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:28.375889 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:28.376039 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.376445 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.377287 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.377653 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.377730 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:28.377739 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:28.377971 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 12:17:28.378009 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 12:17:28.378252 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.375498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42041 (local address 127.0.98.1:39987)
0925 12:17:28.375640 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:28.375666 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:28.375739 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:28.375899 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:28.375900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:28.375902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:28.375970 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:28.375971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.376521 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.376522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:28.377207 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:28.377209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.377786 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.377787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:28.377882 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:28.377885 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:28.378134 (+ 249us) client_negotiation.cc:240] Negotiation successful
0925 12:17:28.378175 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260925 12:17:28.378437 4755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e991a287c5e45cbb277af30e6c44c5d" candidate_uuid: "b595d6b257f840e4936e0745ec98f0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2856cc055204b689dc07b1c8dcafda3" is_pre_election: true
I20260925 12:17:28.378587 4755 raft_consensus.cc:2454] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b595d6b257f840e4936e0745ec98f0c4 in term 0.
I20260925 12:17:28.378543 4888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e991a287c5e45cbb277af30e6c44c5d" candidate_uuid: "b595d6b257f840e4936e0745ec98f0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69302a8c35954e09a702fe35f042889c" is_pre_election: true
I20260925 12:17:28.378684 4888 raft_consensus.cc:2454] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b595d6b257f840e4936e0745ec98f0c4 in term 0.
I20260925 12:17:28.378800 4555 leader_election.cc:302] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b595d6b257f840e4936e0745ec98f0c4, c2856cc055204b689dc07b1c8dcafda3; no voters:
I20260925 12:17:28.378907 4953 raft_consensus.cc:2781] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:17:28.378948 4953 raft_consensus.cc:482] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:17:28.378961 4953 raft_consensus.cc:3037] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:28.379515 4953 raft_consensus.cc:504] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2856cc055204b689dc07b1c8dcafda3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "b595d6b257f840e4936e0745ec98f0c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38619 } } peers { permanent_uuid: "69302a8c35954e09a702fe35f042889c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 12:17:28.379660 4953 leader_election.cc:288] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4 [CANDIDATE]: Term 1 election: Requested vote from peers c2856cc055204b689dc07b1c8dcafda3 (127.0.98.2:44017), 69302a8c35954e09a702fe35f042889c (127.0.98.3:42041)
I20260925 12:17:28.379747 4755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e991a287c5e45cbb277af30e6c44c5d" candidate_uuid: "b595d6b257f840e4936e0745ec98f0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2856cc055204b689dc07b1c8dcafda3"
I20260925 12:17:28.379823 4755 raft_consensus.cc:3037] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:28.380442 4755 raft_consensus.cc:2454] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b595d6b257f840e4936e0745ec98f0c4 in term 1.
I20260925 12:17:28.380551 4888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e991a287c5e45cbb277af30e6c44c5d" candidate_uuid: "b595d6b257f840e4936e0745ec98f0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69302a8c35954e09a702fe35f042889c"
I20260925 12:17:28.380599 4555 leader_election.cc:302] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b595d6b257f840e4936e0745ec98f0c4, c2856cc055204b689dc07b1c8dcafda3; no voters:
I20260925 12:17:28.380630 4888 raft_consensus.cc:3037] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:28.380676 4953 raft_consensus.cc:2781] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:28.380805 4953 raft_consensus.cc:686] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4 [term 1 LEADER]: Becoming Leader. State: Replica: b595d6b257f840e4936e0745ec98f0c4, State: Running, Role: LEADER
I20260925 12:17:28.380890 4953 consensus_queue.cc:237] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2856cc055204b689dc07b1c8dcafda3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "b595d6b257f840e4936e0745ec98f0c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38619 } } peers { permanent_uuid: "69302a8c35954e09a702fe35f042889c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 12:17:28.381431 4888 raft_consensus.cc:2454] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b595d6b257f840e4936e0745ec98f0c4 in term 1.
I20260925 12:17:28.381538 4478 catalog_manager.cc:5495] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b595d6b257f840e4936e0745ec98f0c4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b595d6b257f840e4936e0745ec98f0c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2856cc055204b689dc07b1c8dcafda3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b595d6b257f840e4936e0745ec98f0c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69302a8c35954e09a702fe35f042889c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20260925 12:17:28.451721 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.449580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44318 (local address 127.0.98.1:38619)
0925 12:17:28.449616 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:28.449618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:28.449625 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:17:28.449688 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:28.449689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:28.449709 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:28.449783 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:28.449914 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.450331 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.451018 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:28.451195 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:28.451261 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:28.451418 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:28.451517 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:28.451599 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:28.451615 (+ 16us) server_negotiation.cc:299] Negotiation successful
0925 12:17:28.451652 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:17:28.453436 4755 raft_consensus.cc:1261] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3 [term 1 FOLLOWER]: Refusing update from remote peer b595d6b257f840e4936e0745ec98f0c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:28.453436 4888 raft_consensus.cc:1261] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c [term 1 FOLLOWER]: Refusing update from remote peer b595d6b257f840e4936e0745ec98f0c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:28.453752 4953 consensus_queue.cc:1035] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69302a8c35954e09a702fe35f042889c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:28.453929 4959 consensus_queue.cc:1035] T 9e991a287c5e45cbb277af30e6c44c5d P b595d6b257f840e4936e0745ec98f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2856cc055204b689dc07b1c8dcafda3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:28.456043 4963 mvcc.cc:205] Tried to move back new op lower bound from 7333227104062337024 to 7333227103768645632. Current Snapshot: MvccSnapshot[applied={T|T < 7333227104062337024}]
I20260925 12:17:28.456297 4964 mvcc.cc:205] Tried to move back new op lower bound from 7333227104062337024 to 7333227103768645632. Current Snapshot: MvccSnapshot[applied={T|T < 7333227104062337024}]
I20260925 12:17:28.456866 4962 mvcc.cc:205] Tried to move back new op lower bound from 7333227104062337024 to 7333227103768645632. Current Snapshot: MvccSnapshot[applied={T|T < 7333227104062337024 or (T in {7333227104062337024})}]
I20260925 12:17:28.458302 4478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:28.458384 4478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:28.459110 4478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14c153a337dc48d5b82896ef92f526f8: Sending DeleteTablet for 3 replicas of tablet 9e991a287c5e45cbb277af30e6c44c5d
I20260925 12:17:28.459390 4735 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e991a287c5e45cbb277af30e6c44c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46908
I20260925 12:17:28.459523 4602 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e991a287c5e45cbb277af30e6c44c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44316
I20260925 12:17:28.459604 4868 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e991a287c5e45cbb277af30e6c44c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47766
I20260925 12:17:28.459632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 4540
I20260925 12:17:28.459848 4974 tablet_replica.cc:330] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c: stopping tablet replica
I20260925 12:17:28.459920 4974 raft_consensus.cc:2229] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:28.460000 4974 raft_consensus.cc:2258] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:28.460235 4972 tablet_replica.cc:330] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3: stopping tablet replica
I20260925 12:17:28.460309 4972 raft_consensus.cc:2229] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:28.460383 4972 raft_consensus.cc:2258] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:28.460644 4974 ts_tablet_manager.cc:1905] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:28.461109 4972 ts_tablet_manager.cc:1905] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:28.462297 4974 ts_tablet_manager.cc:1918] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:17:28.462378 4974 log.cc:1196] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/9e991a287c5e45cbb277af30e6c44c5d
I20260925 12:17:28.462642 4974 ts_tablet_manager.cc:1939] T 9e991a287c5e45cbb277af30e6c44c5d P 69302a8c35954e09a702fe35f042889c: Deleting consensus metadata
I20260925 12:17:28.462687 4972 ts_tablet_manager.cc:1918] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:17:28.462742 4972 log.cc:1196] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/9e991a287c5e45cbb277af30e6c44c5d
I20260925 12:17:28.462975 4972 ts_tablet_manager.cc:1939] T 9e991a287c5e45cbb277af30e6c44c5d P c2856cc055204b689dc07b1c8dcafda3: Deleting consensus metadata
I20260925 12:17:28.463073 4465 catalog_manager.cc:4841] TS 69302a8c35954e09a702fe35f042889c (127.0.98.3:42041): tablet 9e991a287c5e45cbb277af30e6c44c5d (table test-table [id=76d63fa799964301bc2718c94267ccb5]) successfully deleted
I20260925 12:17:28.463426 4462 catalog_manager.cc:4841] TS c2856cc055204b689dc07b1c8dcafda3 (127.0.98.2:44017): tablet 9e991a287c5e45cbb277af30e6c44c5d (table test-table [id=76d63fa799964301bc2718c94267ccb5]) successfully deleted
W20260925 12:17:28.465394 4465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38619 (error 108)
I20260925 12:17:28.465571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 4673
I20260925 12:17:28.466188 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.465671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38619 (local address 127.0.0.1:44332)
0925 12:17:28.466100 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:28.466154 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409}
W20260925 12:17:28.466310 4465 catalog_manager.cc:4584] TS b595d6b257f840e4936e0745ec98f0c4 (127.0.98.1:38619): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e991a287c5e45cbb277af30e6c44c5d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38619: connect: Connection refused (error 111)
I20260925 12:17:28.470584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 4806
I20260925 12:17:28.475160 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 4447
2026-09-25T12:17:28Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2100 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T12:17:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:28Z Disabled control of system clock
I20260925 12:17:28.500914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44619
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 12:17:28.576745 4979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:28.576911 4979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:28.576938 4979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:28.577889 4979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:28.577917 4979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:28.577935 4979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:28.577946 4979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:28.577957 4979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:17:28.579111 4979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44619
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:28.579259 4979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:28.579452 4979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:28.583858 4985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:28.583910 4979 server_base.cc:1034] running on GCE node
W20260925 12:17:28.583926 4984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:28.583863 4987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:28.584232 4979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:28.584457 4979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:28.585589 4979 hybrid_clock.cc:648] HybridClock initialized: now 1790338648585579 us; error 27 us; skew 500 ppm
I20260925 12:17:28.586704 4979 webserver.cc:460] Webserver started at http://127.0.98.62:40781/ using document root <none> and password file <none>
I20260925 12:17:28.586931 4979 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:28.586971 4979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:28.587109 4979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:28.588110 4979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "2863af1642984925a4c41ac54800715d"
format_stamp: "Formatted at 2026-09-25 12:17:28 on dist-test-slave-clmx"
server_key: "e507df0dfc6b4d54006ab6478e634628"
server_key_iv: "e5cd57ce824c3ba0ba5285b3483a5a29"
server_key_version: "encryptionkey@0"
I20260925 12:17:28.588441 4979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "2863af1642984925a4c41ac54800715d"
format_stamp: "Formatted at 2026-09-25 12:17:28 on dist-test-slave-clmx"
server_key: "e507df0dfc6b4d54006ab6478e634628"
server_key_iv: "e5cd57ce824c3ba0ba5285b3483a5a29"
server_key_version: "encryptionkey@0"
I20260925 12:17:28.589624 4979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:17:28.590258 4993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:28.590445 4979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:28.590528 4979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "2863af1642984925a4c41ac54800715d"
format_stamp: "Formatted at 2026-09-25 12:17:28 on dist-test-slave-clmx"
server_key: "e507df0dfc6b4d54006ab6478e634628"
server_key_iv: "e5cd57ce824c3ba0ba5285b3483a5a29"
server_key_version: "encryptionkey@0"
I20260925 12:17:28.590596 4979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:28.593719 4979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:28.593954 4979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:28.594074 4979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:28.597671 4979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44619
I20260925 12:17:28.597697 5045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44619 every 8 connection(s)
I20260925 12:17:28.597991 4979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:28.598510 5046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:28.600854 5046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d: Bootstrap starting.
I20260925 12:17:28.601356 5046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:28.601578 5046 log.cc:824] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:28.602183 5046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d: No bootstrap required, opened a new log
I20260925 12:17:28.603602 5046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2863af1642984925a4c41ac54800715d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44619 } }
I20260925 12:17:28.603799 5046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:28.603822 5046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2863af1642984925a4c41ac54800715d, State: Initialized, Role: FOLLOWER
I20260925 12:17:28.603940 5046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2863af1642984925a4c41ac54800715d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44619 } }
I20260925 12:17:28.604015 5046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:28.604048 5046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:28.604099 5046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:28.604411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 4979
I20260925 12:17:28.604513 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:17:28.604601 5046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2863af1642984925a4c41ac54800715d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44619 } }
I20260925 12:17:28.604703 392 external_mini_cluster.cc:1428] Setting key cf2df527d641677e2a409c6da4496c02
I20260925 12:17:28.604712 5046 leader_election.cc:302] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2863af1642984925a4c41ac54800715d; no voters:
I20260925 12:17:28.604887 5046 leader_election.cc:288] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:17:28.604926 5049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:28.605055 5049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d [term 1 LEADER]: Becoming Leader. State: Replica: 2863af1642984925a4c41ac54800715d, State: Running, Role: LEADER
I20260925 12:17:28.605130 5046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:28.605130 5049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2863af1642984925a4c41ac54800715d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44619 } }
I20260925 12:17:28.605468 5049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2863af1642984925a4c41ac54800715d. Latest consensus state: current_term: 1 leader_uuid: "2863af1642984925a4c41ac54800715d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2863af1642984925a4c41ac54800715d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44619 } } }
I20260925 12:17:28.605531 5049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:28.605662 5050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2863af1642984925a4c41ac54800715d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2863af1642984925a4c41ac54800715d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44619 } } }
I20260925 12:17:28.605731 5050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d [sys.catalog]: This master's current role is: LEADER
W20260925 12:17:28.606817 5067 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:17:28.607211 5067 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:17:28.607311 5056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:28.607470 5056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:17:28.607683 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.605299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.0.98.62:44619)
0925 12:17:28.606123 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:28.606132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:28.606167 (+ 35us) server_negotiation.cc:407] Connection header received
0925 12:17:28.606287 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:28.606293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:28.606343 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:28.606485 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:28.607154 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:28.607157 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:28.607157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:28.607161 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:28.607198 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:28.607210 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:28.607266 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:28.607404 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 12:17:28.607457 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
I20260925 12:17:28.608774 5056 catalog_manager.cc:1347] Generated new cluster ID: e5eea295b1224dbe8285de1a3c15c749
I20260925 12:17:28.608822 5056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:17:28.621943 5056 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:17:28.622125 5056 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:28.624837 5056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d: Generated new TSK 0
I20260925 12:17:28.625002 5056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:17:28.633023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44619
--builtin_ntp_servers=127.0.98.20:45929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 12:17:28.712913 5070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:28.713090 5070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:28.713112 5070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:28.714222 5070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:28.714258 5070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:28.714283 5070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:17:28.715565 5070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:28.715835 5070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:28.716109 5070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:28.720739 5075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:28.720739 5078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:28.720727 5076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:28.721333 5070 server_base.cc:1034] running on GCE node
I20260925 12:17:28.721581 5070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:28.721880 5070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:28.723065 5070 hybrid_clock.cc:648] HybridClock initialized: now 1790338648723052 us; error 41 us; skew 500 ppm
I20260925 12:17:28.724336 5070 webserver.cc:460] Webserver started at http://127.0.98.1:40343/ using document root <none> and password file <none>
I20260925 12:17:28.724579 5070 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:28.724632 5070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:28.724748 5070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:28.725968 5070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "3bb9189907f84fc08dca87f7a46fc1da"
format_stamp: "Formatted at 2026-09-25 12:17:28 on dist-test-slave-clmx"
server_key: "93427559b3ded2aa76681aab9ef53be9"
server_key_iv: "29cb06e2c160213d7c6a87be183d11d7"
server_key_version: "encryptionkey@0"
I20260925 12:17:28.726337 5070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3bb9189907f84fc08dca87f7a46fc1da"
format_stamp: "Formatted at 2026-09-25 12:17:28 on dist-test-slave-clmx"
server_key: "93427559b3ded2aa76681aab9ef53be9"
server_key_iv: "29cb06e2c160213d7c6a87be183d11d7"
server_key_version: "encryptionkey@0"
I20260925 12:17:28.727862 5070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:17:28.728636 5084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:28.728825 5070 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:28.728953 5070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "3bb9189907f84fc08dca87f7a46fc1da"
format_stamp: "Formatted at 2026-09-25 12:17:28 on dist-test-slave-clmx"
server_key: "93427559b3ded2aa76681aab9ef53be9"
server_key_iv: "29cb06e2c160213d7c6a87be183d11d7"
server_key_version: "encryptionkey@0"
I20260925 12:17:28.729028 5070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:28.733183 5070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:28.733392 5070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:28.733484 5070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:28.733618 5070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:28.733825 5070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:28.733852 5070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:28.733870 5070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:28.733882 5070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:28.739048 5070 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42851
I20260925 12:17:28.739104 5197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42851 every 8 connection(s)
I20260925 12:17:28.739369 5070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:17:28.741782 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.739975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45457 (local address 127.0.98.62:44619)
0925 12:17:28.740012 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:28.740014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:28.740187 (+ 173us) server_negotiation.cc:407] Connection header received
0925 12:17:28.740569 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:28.740570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:28.740575 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:28.740636 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:28.741635 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:28.741635 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:28.741635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:28.741638 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:28.741647 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:28.741649 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:28.741685 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:28.741728 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 12:17:28.741738 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:17:28.741940 5200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.739953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44619 (local address 127.0.98.1:45457)
0925 12:17:28.740100 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:28.740136 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:28.740486 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:28.740679 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:28.740681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:28.740688 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:28.741529 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:17:28.741529 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:28.741551 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:17:28.741575 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:28.741685 (+ 110us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:28.741687 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:28.741689 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:17:28.741729 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 12:17:28.741768 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":82,"threads_started":1}
I20260925 12:17:28.742403 5198 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44619
I20260925 12:17:28.742470 5198 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:28.742609 5198 heartbeater.cc:511] Master 127.0.98.62:44619 requested a full tablet report, sending...
I20260925 12:17:28.743049 5010 ts_manager.cc:194] Registered new tserver with Master: 3bb9189907f84fc08dca87f7a46fc1da (127.0.98.1:42851)
I20260925 12:17:28.747779 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 5070
I20260925 12:17:28.747875 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:17:28.748040 392 external_mini_cluster.cc:1428] Setting key b9685f7399f4f8805c423081b4df11c3
I20260925 12:17:28.748747 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44619
--builtin_ntp_servers=127.0.98.20:45929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 12:17:28.821616 5201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:28.821779 5201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:28.821798 5201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:28.822937 5201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:28.822973 5201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:28.822999 5201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:17:28.824556 5201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:28.824765 5201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:28.825026 5201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:28.829573 5207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:28.829557 5209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:28.829547 5206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:28.829766 5201 server_base.cc:1034] running on GCE node
I20260925 12:17:28.829988 5201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:28.830186 5201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:28.831331 5201 hybrid_clock.cc:648] HybridClock initialized: now 1790338648831319 us; error 21 us; skew 500 ppm
I20260925 12:17:28.832635 5201 webserver.cc:460] Webserver started at http://127.0.98.2:42907/ using document root <none> and password file <none>
I20260925 12:17:28.832868 5201 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:28.832942 5201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:28.833073 5201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:28.834023 5201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "628c5114b897435998ef33bc1b89d14d"
format_stamp: "Formatted at 2026-09-25 12:17:28 on dist-test-slave-clmx"
server_key: "4c0f7c388fae873b13b2f64fe54baf4f"
server_key_iv: "76cc153fe7ae181f296f87022909cc95"
server_key_version: "encryptionkey@0"
I20260925 12:17:28.834359 5201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "628c5114b897435998ef33bc1b89d14d"
format_stamp: "Formatted at 2026-09-25 12:17:28 on dist-test-slave-clmx"
server_key: "4c0f7c388fae873b13b2f64fe54baf4f"
server_key_iv: "76cc153fe7ae181f296f87022909cc95"
server_key_version: "encryptionkey@0"
I20260925 12:17:28.835520 5201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:17:28.836185 5215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:28.836339 5201 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:28.836427 5201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "628c5114b897435998ef33bc1b89d14d"
format_stamp: "Formatted at 2026-09-25 12:17:28 on dist-test-slave-clmx"
server_key: "4c0f7c388fae873b13b2f64fe54baf4f"
server_key_iv: "76cc153fe7ae181f296f87022909cc95"
server_key_version: "encryptionkey@0"
I20260925 12:17:28.836508 5201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:28.839602 5201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:28.839862 5201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:28.839995 5201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:28.840195 5201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:28.840461 5201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:28.840512 5201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:28.840546 5201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:28.840571 5201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:28.846069 5201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38685
I20260925 12:17:28.846100 5328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38685 every 8 connection(s)
I20260925 12:17:28.846374 5201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:17:28.848835 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.846996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55495 (local address 127.0.98.62:44619)
0925 12:17:28.847047 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:28.847049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:28.847228 (+ 179us) server_negotiation.cc:407] Connection header received
0925 12:17:28.847596 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:28.847597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:28.847602 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:28.847699 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:28.848652 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:28.848653 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:28.848653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:28.848657 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:28.848670 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:28.848673 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:28.848736 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:28.848778 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 12:17:28.848789 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 12:17:28.849005 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.846995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44619 (local address 127.0.98.2:55495)
0925 12:17:28.847131 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:28.847157 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:28.847506 (+ 349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:28.847758 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:28.847761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:28.847774 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:28.848537 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:17:28.848538 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:28.848561 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:17:28.848584 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:28.848732 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:28.848734 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:28.848737 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:28.848783 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 12:17:28.848823 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":62,"threads_started":1}
I20260925 12:17:28.849462 5329 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44619
I20260925 12:17:28.849527 5329 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:28.849676 5329 heartbeater.cc:511] Master 127.0.98.62:44619 requested a full tablet report, sending...
I20260925 12:17:28.850033 5010 ts_manager.cc:194] Registered new tserver with Master: 628c5114b897435998ef33bc1b89d14d (127.0.98.2:38685)
I20260925 12:17:28.852674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 5201
I20260925 12:17:28.852759 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:17:28.852892 392 external_mini_cluster.cc:1428] Setting key 66255612a584ad113998dc65cf618565
I20260925 12:17:28.853642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44619
--builtin_ntp_servers=127.0.98.20:45929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 12:17:28.928793 5332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:28.928969 5332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:28.929003 5332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:28.930018 5332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:28.930049 5332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:28.930074 5332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:17:28.931254 5332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:28.931422 5332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:28.931618 5332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:28.936452 5337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:28.936479 5338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:28.936465 5332 server_base.cc:1034] running on GCE node
W20260925 12:17:28.936452 5340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:28.936828 5332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:28.937006 5332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:28.938133 5332 hybrid_clock.cc:648] HybridClock initialized: now 1790338648938122 us; error 28 us; skew 500 ppm
I20260925 12:17:28.939229 5332 webserver.cc:460] Webserver started at http://127.0.98.3:35017/ using document root <none> and password file <none>
I20260925 12:17:28.939442 5332 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:28.939482 5332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:28.939638 5332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:28.940637 5332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "bf86084de9e1439fabb9c00f315553a6"
format_stamp: "Formatted at 2026-09-25 12:17:28 on dist-test-slave-clmx"
server_key: "6a56324d35bc934ea1097129acd52570"
server_key_iv: "52bfdfe7838ab7b33bb5bf2362ad08eb"
server_key_version: "encryptionkey@0"
I20260925 12:17:28.940999 5332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bf86084de9e1439fabb9c00f315553a6"
format_stamp: "Formatted at 2026-09-25 12:17:28 on dist-test-slave-clmx"
server_key: "6a56324d35bc934ea1097129acd52570"
server_key_iv: "52bfdfe7838ab7b33bb5bf2362ad08eb"
server_key_version: "encryptionkey@0"
I20260925 12:17:28.942191 5332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:17:28.942818 5346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:28.942991 5332 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:28.943080 5332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "bf86084de9e1439fabb9c00f315553a6"
format_stamp: "Formatted at 2026-09-25 12:17:28 on dist-test-slave-clmx"
server_key: "6a56324d35bc934ea1097129acd52570"
server_key_iv: "52bfdfe7838ab7b33bb5bf2362ad08eb"
server_key_version: "encryptionkey@0"
I20260925 12:17:28.943173 5332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:28.946235 5332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:28.946463 5332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:28.946590 5332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:28.946775 5332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:28.947031 5332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:28.947075 5332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:28.947108 5332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:28.947131 5332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:28.952232 5332 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41305
I20260925 12:17:28.952299 5459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41305 every 8 connection(s)
I20260925 12:17:28.952569 5332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:17:28.955204 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.953191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45851 (local address 127.0.98.62:44619)
0925 12:17:28.953266 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:28.953268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:28.953483 (+ 215us) server_negotiation.cc:407] Connection header received
0925 12:17:28.953931 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:28.953932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:28.953939 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:28.954000 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:28.954999 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:28.955000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:28.955000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:28.955005 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:28.955018 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:28.955023 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:28.955064 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:28.955126 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 12:17:28.955139 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 12:17:28.955358 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.953175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44619 (local address 127.0.98.3:45851)
0925 12:17:28.953383 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:28.953409 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:28.953829 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:28.954060 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:28.954064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:28.954073 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:28.954852 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:17:28.954852 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:28.954875 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:17:28.954908 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:28.955069 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:28.955073 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:28.955076 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:28.955135 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 12:17:28.955184 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":101,"threads_started":1}
I20260925 12:17:28.955756 5460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44619
I20260925 12:17:28.955832 5460 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:28.955924 5460 heartbeater.cc:511] Master 127.0.98.62:44619 requested a full tablet report, sending...
I20260925 12:17:28.956187 5010 ts_manager.cc:194] Registered new tserver with Master: bf86084de9e1439fabb9c00f315553a6 (127.0.98.3:41305)
I20260925 12:17:28.957700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 5332
I20260925 12:17:28.957798 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:17:28.957975 392 external_mini_cluster.cc:1428] Setting key 407c18671f96b9648b235b0386ff0f5a
I20260925 12:17:28.958683 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:17:28.960515 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.959739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39012 (local address 127.0.98.62:44619)
0925 12:17:28.959810 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:28.959811 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:28.959907 (+ 96us) server_negotiation.cc:407] Connection header received
0925 12:17:28.959963 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:28.959963 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:28.959967 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:28.960016 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:28.960376 (+ 360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:28.960377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:28.960377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:28.960380 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:28.960392 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:28.960395 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:28.960433 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:28.960466 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 12:17:28.960476 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 12:17:28.961716 5010 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:17:28.963099 5010 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:17:28.966497 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.964967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38685 (local address 127.0.0.1:49088)
0925 12:17:28.965130 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:28.965163 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:28.965231 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:28.965750 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:28.965751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:28.965754 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:28.966087 (+ 333us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:17:28.966088 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:28.966095 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:17:28.966116 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:28.966345 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:28.966346 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:28.966346 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 12:17:28.966409 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 12:17:28.966425 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":50,"threads_started":1}
I20260925 12:17:28.966506 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.965154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49568 (local address 127.0.98.3:41305)
0925 12:17:28.965301 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:28.965305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:28.965420 (+ 115us) server_negotiation.cc:407] Connection header received
0925 12:17:28.965453 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:28.965454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:28.965459 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:28.965514 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:28.966203 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:28.966209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:28.966209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:28.966213 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:28.966239 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:28.966255 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:28.966302 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:28.966408 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 12:17:28.966426 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260925 12:17:28.966516 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.965049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49088 (local address 127.0.98.2:38685)
0925 12:17:28.965632 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:28.965635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:28.965646 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:17:28.965668 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:28.965670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:28.965678 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:28.965747 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:28.966200 (+ 453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:28.966208 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:28.966208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:28.966213 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:28.966243 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:28.966273 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:28.966322 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:28.966426 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 12:17:28.966441 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":72,"threads_started":1}
I20260925 12:17:28.966914 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.964994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41305 (local address 127.0.0.1:49568)
0925 12:17:28.965355 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:28.965364 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:28.965414 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:28.965536 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:28.965541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:28.965547 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:28.966069 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:17:28.966070 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:28.966081 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:17:28.966124 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:28.966323 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:28.966327 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:28.966329 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:17:28.966400 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 12:17:28.966420 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":71,"threads_started":1}
I20260925 12:17:28.967170 5263 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe2d6c006c74b06998e70e5aa264079 (DEFAULT_TABLE table=test-table [id=4bb4f4ebc4e64c65b8d2d3015e54b3e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:28.967473 5263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe2d6c006c74b06998e70e5aa264079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:28.967491 5394 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe2d6c006c74b06998e70e5aa264079 (DEFAULT_TABLE table=test-table [id=4bb4f4ebc4e64c65b8d2d3015e54b3e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:28.967759 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.965004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42851 (local address 127.0.0.1:36510)
0925 12:17:28.965472 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:28.965477 (+ 5us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:28.965527 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:28.966455 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:28.966455 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:28.966457 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:28.967271 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:17:28.967272 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:28.967278 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:17:28.967289 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:28.967571 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:28.967572 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:28.967573 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 12:17:28.967693 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 12:17:28.967706 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":80,"threads_started":1}
I20260925 12:17:28.967795 5394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe2d6c006c74b06998e70e5aa264079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:28.968003 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:28.965198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36510 (local address 127.0.98.1:42851)
0925 12:17:28.965335 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:28.965339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:28.965719 (+ 380us) server_negotiation.cc:407] Connection header received
0925 12:17:28.965746 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:28.965748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:28.965757 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:28.965813 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:28.967371 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:28.967379 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:28.967380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:28.967384 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:28.967431 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:28.967459 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:28.967520 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:28.967896 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 12:17:28.967916 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
I20260925 12:17:28.968707 5132 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe2d6c006c74b06998e70e5aa264079 (DEFAULT_TABLE table=test-table [id=4bb4f4ebc4e64c65b8d2d3015e54b3e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:28.968995 5132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe2d6c006c74b06998e70e5aa264079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:28.970003 5474 tablet_bootstrap.cc:492] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6: Bootstrap starting.
I20260925 12:17:28.970819 5474 tablet_bootstrap.cc:654] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:28.971050 5474 log.cc:824] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:28.971334 5475 tablet_bootstrap.cc:492] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d: Bootstrap starting.
I20260925 12:17:28.971731 5474 tablet_bootstrap.cc:492] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6: No bootstrap required, opened a new log
I20260925 12:17:28.971815 5474 ts_tablet_manager.cc:1397] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:17:28.971873 5477 tablet_bootstrap.cc:492] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da: Bootstrap starting.
I20260925 12:17:28.972517 5477 tablet_bootstrap.cc:654] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:28.972851 5477 log.cc:824] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:28.973569 5477 tablet_bootstrap.cc:492] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da: No bootstrap required, opened a new log
I20260925 12:17:28.973634 5477 ts_tablet_manager.cc:1397] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:17:28.974453 5474 raft_consensus.cc:348] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628c5114b897435998ef33bc1b89d14d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38685 } } peers { permanent_uuid: "bf86084de9e1439fabb9c00f315553a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "3bb9189907f84fc08dca87f7a46fc1da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42851 } }
I20260925 12:17:28.974609 5474 raft_consensus.cc:374] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:28.974644 5474 raft_consensus.cc:729] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf86084de9e1439fabb9c00f315553a6, State: Initialized, Role: FOLLOWER
I20260925 12:17:28.974752 5474 consensus_queue.cc:260] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628c5114b897435998ef33bc1b89d14d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38685 } } peers { permanent_uuid: "bf86084de9e1439fabb9c00f315553a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "3bb9189907f84fc08dca87f7a46fc1da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42851 } }
I20260925 12:17:28.974996 5474 ts_tablet_manager.cc:1428] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:17:28.975023 5460 heartbeater.cc:503] Master 127.0.98.62:44619 was elected leader, sending a full tablet report...
I20260925 12:17:28.975603 5475 tablet_bootstrap.cc:654] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:28.975982 5475 log.cc:824] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:28.976274 5477 raft_consensus.cc:348] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628c5114b897435998ef33bc1b89d14d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38685 } } peers { permanent_uuid: "bf86084de9e1439fabb9c00f315553a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "3bb9189907f84fc08dca87f7a46fc1da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42851 } }
I20260925 12:17:28.976444 5477 raft_consensus.cc:374] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:28.976483 5477 raft_consensus.cc:729] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb9189907f84fc08dca87f7a46fc1da, State: Initialized, Role: FOLLOWER
I20260925 12:17:28.976588 5477 consensus_queue.cc:260] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628c5114b897435998ef33bc1b89d14d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38685 } } peers { permanent_uuid: "bf86084de9e1439fabb9c00f315553a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "3bb9189907f84fc08dca87f7a46fc1da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42851 } }
I20260925 12:17:28.976629 5475 tablet_bootstrap.cc:492] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d: No bootstrap required, opened a new log
I20260925 12:17:28.976678 5475 ts_tablet_manager.cc:1397] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 12:17:28.976819 5477 ts_tablet_manager.cc:1428] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:17:28.976873 5198 heartbeater.cc:503] Master 127.0.98.62:44619 was elected leader, sending a full tablet report...
I20260925 12:17:28.978699 5475 raft_consensus.cc:348] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628c5114b897435998ef33bc1b89d14d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38685 } } peers { permanent_uuid: "bf86084de9e1439fabb9c00f315553a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "3bb9189907f84fc08dca87f7a46fc1da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42851 } }
I20260925 12:17:28.978803 5475 raft_consensus.cc:374] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:28.978821 5475 raft_consensus.cc:729] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628c5114b897435998ef33bc1b89d14d, State: Initialized, Role: FOLLOWER
I20260925 12:17:28.978938 5475 consensus_queue.cc:260] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628c5114b897435998ef33bc1b89d14d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38685 } } peers { permanent_uuid: "bf86084de9e1439fabb9c00f315553a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "3bb9189907f84fc08dca87f7a46fc1da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42851 } }
I20260925 12:17:28.979157 5475 ts_tablet_manager.cc:1428] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 12:17:28.979174 5329 heartbeater.cc:503] Master 127.0.98.62:44619 was elected leader, sending a full tablet report...
W20260925 12:17:28.989799 5199 tablet.cc:2369] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:17:29.040004 5481 raft_consensus.cc:482] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:17:29.040110 5481 raft_consensus.cc:504] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628c5114b897435998ef33bc1b89d14d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38685 } } peers { permanent_uuid: "bf86084de9e1439fabb9c00f315553a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "3bb9189907f84fc08dca87f7a46fc1da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42851 } }
I20260925 12:17:29.040462 5481 leader_election.cc:288] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 628c5114b897435998ef33bc1b89d14d (127.0.98.2:38685), bf86084de9e1439fabb9c00f315553a6 (127.0.98.3:41305)
I20260925 12:17:29.041613 5200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:29.040597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38685 (local address 127.0.98.1:41193)
0925 12:17:29.040645 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:29.040655 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:29.040688 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:29.040815 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:29.040816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:29.040834 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:29.041261 (+ 427us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:17:29.041262 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:29.041267 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:17:29.041277 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:29.041466 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:29.041466 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:29.041467 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 12:17:29.041517 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 12:17:29.041530 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 12:17:29.041632 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:29.040669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41193 (local address 127.0.98.2:38685)
0925 12:17:29.040705 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:29.040707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:29.040715 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:29.040737 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:29.040738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:29.040743 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:29.040791 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:29.041350 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:29.041351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:29.041351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:29.041355 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:29.041369 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:29.041373 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:29.041456 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:29.041542 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 12:17:29.041556 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:17:29.041740 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:29.040690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48351 (local address 127.0.98.3:41305)
0925 12:17:29.040740 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:29.040742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:29.040847 (+ 105us) server_negotiation.cc:407] Connection header received
0925 12:17:29.040895 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:29.040897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:29.040901 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:29.040961 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:29.041538 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:29.041539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:29.041540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:29.041543 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:29.041555 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:29.041559 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:29.041608 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:29.041650 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 12:17:29.041661 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 12:17:29.041749 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:29.040625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41305 (local address 127.0.98.1:48351)
0925 12:17:29.040781 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:29.040789 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:29.040839 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:29.040988 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:29.040988 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:29.040990 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:29.041432 (+ 442us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:17:29.041432 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:29.041439 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:17:29.041453 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:29.041605 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:29.041606 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:29.041606 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 12:17:29.041651 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 12:17:29.041664 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
I20260925 12:17:29.041903 5283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2d6c006c74b06998e70e5aa264079" candidate_uuid: "3bb9189907f84fc08dca87f7a46fc1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "628c5114b897435998ef33bc1b89d14d" is_pre_election: true
I20260925 12:17:29.042073 5283 raft_consensus.cc:2454] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bb9189907f84fc08dca87f7a46fc1da in term 0.
I20260925 12:17:29.042356 5088 leader_election.cc:302] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bb9189907f84fc08dca87f7a46fc1da, 628c5114b897435998ef33bc1b89d14d; no voters:
I20260925 12:17:29.042488 5481 raft_consensus.cc:2781] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:17:29.042495 5414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2d6c006c74b06998e70e5aa264079" candidate_uuid: "3bb9189907f84fc08dca87f7a46fc1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf86084de9e1439fabb9c00f315553a6" is_pre_election: true
I20260925 12:17:29.042541 5481 raft_consensus.cc:482] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:17:29.042585 5481 raft_consensus.cc:3037] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:29.042631 5414 raft_consensus.cc:2454] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bb9189907f84fc08dca87f7a46fc1da in term 0.
I20260925 12:17:29.043311 5481 raft_consensus.cc:504] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628c5114b897435998ef33bc1b89d14d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38685 } } peers { permanent_uuid: "bf86084de9e1439fabb9c00f315553a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "3bb9189907f84fc08dca87f7a46fc1da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42851 } }
I20260925 12:17:29.043432 5481 leader_election.cc:288] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da [CANDIDATE]: Term 1 election: Requested vote from peers 628c5114b897435998ef33bc1b89d14d (127.0.98.2:38685), bf86084de9e1439fabb9c00f315553a6 (127.0.98.3:41305)
I20260925 12:17:29.043548 5414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2d6c006c74b06998e70e5aa264079" candidate_uuid: "3bb9189907f84fc08dca87f7a46fc1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf86084de9e1439fabb9c00f315553a6"
I20260925 12:17:29.043563 5283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2d6c006c74b06998e70e5aa264079" candidate_uuid: "3bb9189907f84fc08dca87f7a46fc1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "628c5114b897435998ef33bc1b89d14d"
I20260925 12:17:29.043656 5283 raft_consensus.cc:3037] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:29.043651 5414 raft_consensus.cc:3037] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:29.044345 5283 raft_consensus.cc:2454] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb9189907f84fc08dca87f7a46fc1da in term 1.
I20260925 12:17:29.044354 5414 raft_consensus.cc:2454] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb9189907f84fc08dca87f7a46fc1da in term 1.
I20260925 12:17:29.044515 5087 leader_election.cc:302] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bb9189907f84fc08dca87f7a46fc1da, bf86084de9e1439fabb9c00f315553a6; no voters:
I20260925 12:17:29.044625 5481 raft_consensus.cc:2781] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:29.044788 5481 raft_consensus.cc:686] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da [term 1 LEADER]: Becoming Leader. State: Replica: 3bb9189907f84fc08dca87f7a46fc1da, State: Running, Role: LEADER
I20260925 12:17:29.044907 5481 consensus_queue.cc:237] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628c5114b897435998ef33bc1b89d14d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38685 } } peers { permanent_uuid: "bf86084de9e1439fabb9c00f315553a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "3bb9189907f84fc08dca87f7a46fc1da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42851 } }
I20260925 12:17:29.045696 5010 catalog_manager.cc:5495] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bb9189907f84fc08dca87f7a46fc1da (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3bb9189907f84fc08dca87f7a46fc1da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628c5114b897435998ef33bc1b89d14d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf86084de9e1439fabb9c00f315553a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb9189907f84fc08dca87f7a46fc1da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42851 } health_report { overall_health: HEALTHY } } }
I20260925 12:17:29.058177 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:29.057431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.0.98.1:42851)
0925 12:17:29.057479 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:29.057481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:29.057487 (+ 6us) server_negotiation.cc:407] Connection header received
0925 12:17:29.057547 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:29.057547 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:29.057555 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:29.057600 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:29.057985 (+ 385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:29.057986 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:29.057986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:29.057989 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:29.058000 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:29.058003 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:29.058047 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:29.058075 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 12:17:29.058084 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 12:17:29.059670 5283 raft_consensus.cc:1261] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d [term 1 FOLLOWER]: Refusing update from remote peer 3bb9189907f84fc08dca87f7a46fc1da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:29.059700 5414 raft_consensus.cc:1261] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6 [term 1 FOLLOWER]: Refusing update from remote peer 3bb9189907f84fc08dca87f7a46fc1da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:29.060063 5484 consensus_queue.cc:1035] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf86084de9e1439fabb9c00f315553a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:29.060757 5484 consensus_queue.cc:1035] T 9fe2d6c006c74b06998e70e5aa264079 P 3bb9189907f84fc08dca87f7a46fc1da [LEADER]: Connected to new peer: Peer: permanent_uuid: "628c5114b897435998ef33bc1b89d14d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:29.062333 5487 mvcc.cc:205] Tried to move back new op lower bound from 7333227106546159616 to 7333227106488532992. Current Snapshot: MvccSnapshot[applied={T|T < 7333227106546159616}]
I20260925 12:17:29.063097 5488 mvcc.cc:205] Tried to move back new op lower bound from 7333227106546159616 to 7333227106488532992. Current Snapshot: MvccSnapshot[applied={T|T < 7333227106546159616}]
I20260925 12:17:29.063105 5490 mvcc.cc:205] Tried to move back new op lower bound from 7333227106546159616 to 7333227106488532992. Current Snapshot: MvccSnapshot[applied={T|T < 7333227106546159616}]
I20260925 12:17:29.064436 5009 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:29.064520 5009 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:29.065178 5009 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2863af1642984925a4c41ac54800715d: Sending DeleteTablet for 3 replicas of tablet 9fe2d6c006c74b06998e70e5aa264079
I20260925 12:17:29.065498 5263 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe2d6c006c74b06998e70e5aa264079 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:29 UTC) from {username='slave'} at 127.0.0.1:49088
I20260925 12:17:29.065557 5132 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe2d6c006c74b06998e70e5aa264079 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:29 UTC) from {username='slave'} at 127.0.0.1:36510
I20260925 12:17:29.065726 5497 tablet_replica.cc:330] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d: stopping tablet replica
I20260925 12:17:29.065781 5394 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe2d6c006c74b06998e70e5aa264079 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:29 UTC) from {username='slave'} at 127.0.0.1:49568
I20260925 12:17:29.065822 5497 raft_consensus.cc:2229] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:29.065922 5497 raft_consensus.cc:2258] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:29.066021 5499 tablet_replica.cc:330] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6: stopping tablet replica
I20260925 12:17:29.066141 5499 raft_consensus.cc:2229] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:29.066227 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 5070
I20260925 12:17:29.066808 5499 raft_consensus.cc:2258] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:29.067180 5499 ts_tablet_manager.cc:1905] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:29.068550 5497 ts_tablet_manager.cc:1905] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:29.068565 5499 ts_tablet_manager.cc:1918] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:17:29.068622 5499 log.cc:1196] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/9fe2d6c006c74b06998e70e5aa264079
I20260925 12:17:29.068850 5499 ts_tablet_manager.cc:1939] T 9fe2d6c006c74b06998e70e5aa264079 P bf86084de9e1439fabb9c00f315553a6: Deleting consensus metadata
I20260925 12:17:29.069205 4996 catalog_manager.cc:4841] TS bf86084de9e1439fabb9c00f315553a6 (127.0.98.3:41305): tablet 9fe2d6c006c74b06998e70e5aa264079 (table test-table [id=4bb4f4ebc4e64c65b8d2d3015e54b3e5]) successfully deleted
I20260925 12:17:29.069888 5497 ts_tablet_manager.cc:1918] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:17:29.069947 5497 log.cc:1196] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/9fe2d6c006c74b06998e70e5aa264079
I20260925 12:17:29.070176 5497 ts_tablet_manager.cc:1939] T 9fe2d6c006c74b06998e70e5aa264079 P 628c5114b897435998ef33bc1b89d14d: Deleting consensus metadata
I20260925 12:17:29.070528 4997 catalog_manager.cc:4841] TS 628c5114b897435998ef33bc1b89d14d (127.0.98.2:38685): tablet 9fe2d6c006c74b06998e70e5aa264079 (table test-table [id=4bb4f4ebc4e64c65b8d2d3015e54b3e5]) successfully deleted
W20260925 12:17:29.071408 4995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42851 (error 108)
I20260925 12:17:29.071861 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:29.071762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42851 (local address 127.0.0.1:36530)
0925 12:17:29.071808 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:29.071841 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 12:17:29.071915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 5201
W20260925 12:17:29.074024 4995 catalog_manager.cc:4584] TS 3bb9189907f84fc08dca87f7a46fc1da (127.0.98.1:42851): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9fe2d6c006c74b06998e70e5aa264079: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42851: connect: Connection refused (error 111)
I20260925 12:17:29.076789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 5332
I20260925 12:17:29.080462 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 4979
2026-09-25T12:17:29Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (604 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 12:17:29.099480 392 mini_oidc.cc:152] Starting JWKS server
I20260925 12:17:29.100553 392 webserver.cc:460] Webserver started at https://127.0.0.1:44913/ using document root <none> and password file <none>
I20260925 12:17:29.100647 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 12:17:29.100818 392 webserver.cc:460] Webserver started at http://127.0.0.1:45909/ using document root <none> and password file <none>
2026-09-25T12:17:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:29Z Disabled control of system clock
I20260925 12:17:29.107695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41393
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41393
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44913/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338625091267-392-0/testchainca.cert with env {}
W20260925 12:17:29.180754 5506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:29.180948 5506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:29.180985 5506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:29.182423 5506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:29.182462 5506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:29.182475 5506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:29.182487 5506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:29.182499 5506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 12:17:29.182511 5506 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 12:17:29.182523 5506 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44913/jwks.json
I20260925 12:17:29.183991 5506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44913/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338625091267-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:29.184182 5506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:29.184321 5506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:29.188691 5512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:29.188704 5514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:29.188704 5511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:29.188916 5506 server_base.cc:1034] running on GCE node
I20260925 12:17:29.189196 5506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:29.189445 5506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:29.190572 5506 hybrid_clock.cc:648] HybridClock initialized: now 1790338649190562 us; error 25 us; skew 500 ppm
I20260925 12:17:29.191776 5506 webserver.cc:460] Webserver started at http://127.0.98.62:42031/ using document root <none> and password file <none>
I20260925 12:17:29.192029 5506 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:29.192076 5506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:29.192215 5506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:29.193137 5506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "3dd5d1a91bf641acb810d476a6ecbea4"
format_stamp: "Formatted at 2026-09-25 12:17:29 on dist-test-slave-clmx"
server_key: "d5d1089db04d3bd27ee924962ef5fd91"
server_key_iv: "c85576f31e4ac6acd1891c0d5b212f2a"
server_key_version: "encryptionkey@0"
I20260925 12:17:29.193460 5506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "3dd5d1a91bf641acb810d476a6ecbea4"
format_stamp: "Formatted at 2026-09-25 12:17:29 on dist-test-slave-clmx"
server_key: "d5d1089db04d3bd27ee924962ef5fd91"
server_key_iv: "c85576f31e4ac6acd1891c0d5b212f2a"
server_key_version: "encryptionkey@0"
I20260925 12:17:29.194621 5506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:17:29.195284 5520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:29.195434 5506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:29.195524 5506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "3dd5d1a91bf641acb810d476a6ecbea4"
format_stamp: "Formatted at 2026-09-25 12:17:29 on dist-test-slave-clmx"
server_key: "d5d1089db04d3bd27ee924962ef5fd91"
server_key_iv: "c85576f31e4ac6acd1891c0d5b212f2a"
server_key_version: "encryptionkey@0"
I20260925 12:17:29.195598 5506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:29.212271 5506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:29.212518 5506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:29.212657 5506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:29.216180 5506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41393
I20260925 12:17:29.216212 5572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41393 every 8 connection(s)
I20260925 12:17:29.216496 5506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:29.217335 5573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:29.219271 5573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4: Bootstrap starting.
I20260925 12:17:29.219820 5573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:29.220103 5573 log.cc:824] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:29.220690 5573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4: No bootstrap required, opened a new log
I20260925 12:17:29.221567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 5506
I20260925 12:17:29.221668 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:17:29.221855 392 external_mini_cluster.cc:1428] Setting key fffb22b79a6711f854c30ebc04dfd7bb
I20260925 12:17:29.222714 5573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd5d1a91bf641acb810d476a6ecbea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41393 } }
I20260925 12:17:29.222859 5573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:29.222903 5573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd5d1a91bf641acb810d476a6ecbea4, State: Initialized, Role: FOLLOWER
I20260925 12:17:29.223045 5573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd5d1a91bf641acb810d476a6ecbea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41393 } }
I20260925 12:17:29.223127 5573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:29.223160 5573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:29.223222 5573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:29.223908 5573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd5d1a91bf641acb810d476a6ecbea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41393 } }
I20260925 12:17:29.224005 5573 leader_election.cc:302] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dd5d1a91bf641acb810d476a6ecbea4; no voters:
I20260925 12:17:29.224152 5573 leader_election.cc:288] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:17:29.224200 5578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:29.224320 5578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4 [term 1 LEADER]: Becoming Leader. State: Replica: 3dd5d1a91bf641acb810d476a6ecbea4, State: Running, Role: LEADER
I20260925 12:17:29.224354 5573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:29.224404 5578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd5d1a91bf641acb810d476a6ecbea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41393 } }
I20260925 12:17:29.224689 5578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dd5d1a91bf641acb810d476a6ecbea4. Latest consensus state: current_term: 1 leader_uuid: "3dd5d1a91bf641acb810d476a6ecbea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd5d1a91bf641acb810d476a6ecbea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41393 } } }
I20260925 12:17:29.224776 5578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:29.224695 5580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dd5d1a91bf641acb810d476a6ecbea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd5d1a91bf641acb810d476a6ecbea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41393 } } }
I20260925 12:17:29.224860 5580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4 [sys.catalog]: This master's current role is: LEADER
W20260925 12:17:29.226017 5594 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:17:29.226068 5594 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:17:29.226114 5585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:29.226236 5585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:17:29.227495 5585 catalog_manager.cc:1347] Generated new cluster ID: 282e15ef1ea44a0cabb118c48d4b7409
I20260925 12:17:29.227543 5585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:17:29.228048 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:29.222311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38722 (local address 127.0.98.62:41393)
0925 12:17:29.222501 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:29.222509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:29.222540 (+ 31us) server_negotiation.cc:407] Connection header received
0925 12:17:29.222682 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:29.222687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:29.222734 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:29.222862 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:29.223063 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.224799 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.225981 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.226479 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.226604 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:29.227172 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:29.227177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:29.227177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:29.227340 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:29.227368 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:29.227378 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:29.227729 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:29.227764 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 12:17:29.227837 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260925 12:17:29.228745 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:17:29.252728 5585 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:17:29.253364 5585 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:29.259123 5585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dd5d1a91bf641acb810d476a6ecbea4: Generated new TSK 0
I20260925 12:17:29.259276 5585 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 12:17:29.283689 5603 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 12:17:29.283767 5603 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 12:17:29.283951 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:29.270105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38730 (local address 127.0.98.62:41393)
0925 12:17:29.270149 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:29.270151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:29.270224 (+ 73us) server_negotiation.cc:407] Connection header received
0925 12:17:29.270317 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:29.270318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:29.270326 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:29.270371 (+ 45us) server_negotiation.cc:226] Negotiated authn=JWT
0925 12:17:29.270514 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.270949 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.272145 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.272318 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.272369 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:29.272580 (+ 211us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 12:17:29.283785 (+ 11205us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44913/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 12:17:29.283913 (+ 128us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38730: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44913/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 12:17:29.284008 5577 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38730: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44913/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 12:17:29.284332 5602 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:29.270015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41393 (local address 127.0.0.1:38730)
0925 12:17:29.270172 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:29.270182 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:29.270243 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:29.270384 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:29.270385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:29.270387 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 12:17:29.270460 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:29.270462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.271037 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.271038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:29.272067 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:29.272071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.272386 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.272392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:29.272464 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:29.272484 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 12:17:29.283901 (+ 11417us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44913/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 12:17:29.284004 (+ 103us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41393: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44913/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:41393: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44913/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 12:17:29.284803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 5506
2026-09-25T12:17:29Z chronyd exiting
I20260925 12:17:29.319898 392 test_util.cc:175] -----------------------------------------------
I20260925 12:17:29.319952 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338625091267-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (221 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 12:17:29.320622 392 mini_oidc.cc:152] Starting JWKS server
I20260925 12:17:29.321272 392 webserver.cc:460] Webserver started at https://127.0.0.1:44001/ using document root <none> and password file <none>
I20260925 12:17:29.321306 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 12:17:29.321477 392 webserver.cc:460] Webserver started at http://127.0.0.1:39179/ using document root <none> and password file <none>
2026-09-25T12:17:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:29Z Disabled control of system clock
I20260925 12:17:29.328290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38929
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38929
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44001/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338625091267-392-0/testchainca.cert with env {}
W20260925 12:17:29.401037 5609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:29.401245 5609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:29.401280 5609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:29.402370 5609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:29.402444 5609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:29.402468 5609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:29.402496 5609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:29.402529 5609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 12:17:29.402550 5609 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 12:17:29.402571 5609 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44001/jwks.json
I20260925 12:17:29.404042 5609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38929
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44001/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338625091267-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:29.404340 5609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:29.404539 5609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:29.408787 5614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:29.408838 5609 server_base.cc:1034] running on GCE node
W20260925 12:17:29.408790 5615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:29.408797 5617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:29.409147 5609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:29.409406 5609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:29.410565 5609 hybrid_clock.cc:648] HybridClock initialized: now 1790338649410560 us; error 33 us; skew 500 ppm
I20260925 12:17:29.411832 5609 webserver.cc:460] Webserver started at http://127.0.98.62:42749/ using document root <none> and password file <none>
I20260925 12:17:29.412086 5609 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:29.412156 5609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:29.412281 5609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:29.413223 5609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "cb81b9e9263d4a749e2bc97a01947e61"
format_stamp: "Formatted at 2026-09-25 12:17:29 on dist-test-slave-clmx"
server_key: "2e6cddae0af550b713929ff6b70182e5"
server_key_iv: "9ce4fdcf6af4d5303fbf5f77976cdab1"
server_key_version: "encryptionkey@0"
I20260925 12:17:29.413530 5609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "cb81b9e9263d4a749e2bc97a01947e61"
format_stamp: "Formatted at 2026-09-25 12:17:29 on dist-test-slave-clmx"
server_key: "2e6cddae0af550b713929ff6b70182e5"
server_key_iv: "9ce4fdcf6af4d5303fbf5f77976cdab1"
server_key_version: "encryptionkey@0"
I20260925 12:17:29.414671 5609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:17:29.415270 5623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:29.415423 5609 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:29.415512 5609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "cb81b9e9263d4a749e2bc97a01947e61"
format_stamp: "Formatted at 2026-09-25 12:17:29 on dist-test-slave-clmx"
server_key: "2e6cddae0af550b713929ff6b70182e5"
server_key_iv: "9ce4fdcf6af4d5303fbf5f77976cdab1"
server_key_version: "encryptionkey@0"
I20260925 12:17:29.415588 5609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:29.437340 5609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:29.437575 5609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:29.437712 5609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:29.440846 5609 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38929
I20260925 12:17:29.440886 5675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38929 every 8 connection(s)
I20260925 12:17:29.441180 5609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:29.441987 5676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:29.442065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 5609
I20260925 12:17:29.442179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:17:29.442358 392 external_mini_cluster.cc:1428] Setting key 0446f78420df7a9d39b8b5dc9d2ba8cf
I20260925 12:17:29.444619 5676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61: Bootstrap starting.
I20260925 12:17:29.445140 5676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:29.445402 5676 log.cc:824] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:29.445979 5676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61: No bootstrap required, opened a new log
I20260925 12:17:29.447127 5679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:29.442783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48152 (local address 127.0.98.62:38929)
0925 12:17:29.442978 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:29.442986 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:29.443018 (+ 32us) server_negotiation.cc:407] Connection header received
0925 12:17:29.443395 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:29.443400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:29.443449 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:29.443583 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:29.443779 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.444578 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.445349 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.445753 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.445895 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:29.446529 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:29.446533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:29.446533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:29.446687 (+ 154us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:29.446713 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:29.446723 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:29.446791 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:29.446887 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 12:17:29.446956 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260925 12:17:29.448078 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:17:29.447992 5676 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb81b9e9263d4a749e2bc97a01947e61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38929 } }
I20260925 12:17:29.448137 5676 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:29.448179 5676 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb81b9e9263d4a749e2bc97a01947e61, State: Initialized, Role: FOLLOWER
I20260925 12:17:29.448282 5676 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb81b9e9263d4a749e2bc97a01947e61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38929 } }
I20260925 12:17:29.448372 5676 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:29.448412 5676 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:29.448465 5676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:29.449048 5676 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb81b9e9263d4a749e2bc97a01947e61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38929 } }
I20260925 12:17:29.449141 5676 leader_election.cc:302] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb81b9e9263d4a749e2bc97a01947e61; no voters:
I20260925 12:17:29.449271 5676 leader_election.cc:288] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:17:29.449301 5681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:29.449438 5681 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61 [term 1 LEADER]: Becoming Leader. State: Replica: cb81b9e9263d4a749e2bc97a01947e61, State: Running, Role: LEADER
I20260925 12:17:29.449491 5676 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:29.449509 5681 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb81b9e9263d4a749e2bc97a01947e61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38929 } }
I20260925 12:17:29.449786 5682 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb81b9e9263d4a749e2bc97a01947e61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb81b9e9263d4a749e2bc97a01947e61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38929 } } }
I20260925 12:17:29.449884 5682 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:29.450066 5687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:29.449782 5683 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb81b9e9263d4a749e2bc97a01947e61. Latest consensus state: current_term: 1 leader_uuid: "cb81b9e9263d4a749e2bc97a01947e61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb81b9e9263d4a749e2bc97a01947e61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38929 } } }
I20260925 12:17:29.450424 5683 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:29.450654 5687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:17:29.451995 5687 catalog_manager.cc:1347] Generated new cluster ID: e34ae57b49874f32833e946e97a20b54
I20260925 12:17:29.452071 5687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:17:29.461810 5687 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:17:29.462471 5687 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:29.470180 5687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb81b9e9263d4a749e2bc97a01947e61: Generated new TSK 0
I20260925 12:17:29.470317 5687 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 12:17:29.493218 5706 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 12:17:29.493286 5706 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 12:17:29.493566 5679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:29.479581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48166 (local address 127.0.98.62:38929)
0925 12:17:29.479666 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:29.479667 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:29.479732 (+ 65us) server_negotiation.cc:407] Connection header received
0925 12:17:29.479795 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:29.479795 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:29.479805 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:29.479856 (+ 51us) server_negotiation.cc:226] Negotiated authn=JWT
0925 12:17:29.479985 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.480409 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.481510 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.481685 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.481732 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:29.481896 (+ 164us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 12:17:29.493408 (+ 11512us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 12:17:29.493519 (+ 111us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48166: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 12:17:29.493628 5679 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48166: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 12:17:29.493724 5705 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:29.479498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38929 (local address 127.0.0.1:48166)
0925 12:17:29.479664 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:29.479675 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:29.479735 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:29.479869 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:29.479870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:29.479872 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 12:17:29.479927 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:29.479929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.480472 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.480473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:29.481430 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:29.481433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.481750 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.481757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:29.481814 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:29.481820 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 12:17:29.493517 (+ 11697us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 12:17:29.493623 (+ 106us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38929: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260925 12:17:29.494011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 5609
2026-09-25T12:17:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (210 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 12:17:29.530882 392 mini_oidc.cc:152] Starting JWKS server
I20260925 12:17:29.531484 392 webserver.cc:460] Webserver started at https://127.0.0.1:43511/ using document root <none> and password file <none>
I20260925 12:17:29.531519 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 12:17:29.531664 392 webserver.cc:460] Webserver started at http://127.0.0.1:44199/ using document root <none> and password file <none>
2026-09-25T12:17:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:29Z Disabled control of system clock
I20260925 12:17:29.538574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37155
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37155
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43511/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 12:17:29.614576 5712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:29.614784 5712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:29.614820 5712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:29.615957 5712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:29.616019 5712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:29.616039 5712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:29.616060 5712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:29.616088 5712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 12:17:29.616110 5712 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 12:17:29.616133 5712 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43511/jwks.json
I20260925 12:17:29.617404 5712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37155
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43511/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:29.617676 5712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:29.617960 5712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:29.622426 5717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:29.622545 5712 server_base.cc:1034] running on GCE node
W20260925 12:17:29.622449 5720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:29.622445 5718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:29.622855 5712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:29.623140 5712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:29.624272 5712 hybrid_clock.cc:648] HybridClock initialized: now 1790338649624261 us; error 38 us; skew 500 ppm
I20260925 12:17:29.625469 5712 webserver.cc:460] Webserver started at http://127.0.98.62:38559/ using document root <none> and password file <none>
I20260925 12:17:29.625699 5712 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:29.625743 5712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:29.625905 5712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:29.626832 5712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "d08378e8dead4c2a917ae0f0ce652e62"
format_stamp: "Formatted at 2026-09-25 12:17:29 on dist-test-slave-clmx"
server_key: "ac7318b392a78e32b518e0727e2eb597"
server_key_iv: "40a67588632e938c5b5186af0adf1b6a"
server_key_version: "encryptionkey@0"
I20260925 12:17:29.627147 5712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "d08378e8dead4c2a917ae0f0ce652e62"
format_stamp: "Formatted at 2026-09-25 12:17:29 on dist-test-slave-clmx"
server_key: "ac7318b392a78e32b518e0727e2eb597"
server_key_iv: "40a67588632e938c5b5186af0adf1b6a"
server_key_version: "encryptionkey@0"
I20260925 12:17:29.628373 5712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:17:29.628974 5726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:29.629149 5712 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:17:29.629266 5712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "d08378e8dead4c2a917ae0f0ce652e62"
format_stamp: "Formatted at 2026-09-25 12:17:29 on dist-test-slave-clmx"
server_key: "ac7318b392a78e32b518e0727e2eb597"
server_key_iv: "40a67588632e938c5b5186af0adf1b6a"
server_key_version: "encryptionkey@0"
I20260925 12:17:29.629340 5712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:29.644544 5712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:29.644799 5712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:29.644944 5712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:29.648540 5712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37155
I20260925 12:17:29.648568 5778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37155 every 8 connection(s)
I20260925 12:17:29.648859 5712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:29.649468 5779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:29.651784 5779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62: Bootstrap starting.
I20260925 12:17:29.652266 5779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:29.652493 5779 log.cc:824] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:29.652740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 5712
I20260925 12:17:29.652822 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:17:29.653002 392 external_mini_cluster.cc:1428] Setting key 86593299b88da4189f32ca5854049fbd
I20260925 12:17:29.653287 5779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62: No bootstrap required, opened a new log
I20260925 12:17:29.654804 5779 raft_consensus.cc:348] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d08378e8dead4c2a917ae0f0ce652e62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37155 } }
I20260925 12:17:29.654913 5779 raft_consensus.cc:374] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:29.654932 5779 raft_consensus.cc:729] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d08378e8dead4c2a917ae0f0ce652e62, State: Initialized, Role: FOLLOWER
I20260925 12:17:29.655074 5779 consensus_queue.cc:260] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d08378e8dead4c2a917ae0f0ce652e62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37155 } }
I20260925 12:17:29.655149 5779 raft_consensus.cc:388] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:29.655181 5779 raft_consensus.cc:482] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:29.655226 5779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:29.655726 5779 raft_consensus.cc:504] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d08378e8dead4c2a917ae0f0ce652e62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37155 } }
I20260925 12:17:29.655802 5779 leader_election.cc:302] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d08378e8dead4c2a917ae0f0ce652e62; no voters:
I20260925 12:17:29.656005 5779 leader_election.cc:288] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:17:29.656036 5784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:29.656159 5784 raft_consensus.cc:686] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62 [term 1 LEADER]: Becoming Leader. State: Replica: d08378e8dead4c2a917ae0f0ce652e62, State: Running, Role: LEADER
I20260925 12:17:29.656229 5779 sys_catalog.cc:564] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:29.656234 5784 consensus_queue.cc:237] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d08378e8dead4c2a917ae0f0ce652e62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37155 } }
I20260925 12:17:29.656519 5784 sys_catalog.cc:455] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d08378e8dead4c2a917ae0f0ce652e62. Latest consensus state: current_term: 1 leader_uuid: "d08378e8dead4c2a917ae0f0ce652e62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d08378e8dead4c2a917ae0f0ce652e62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37155 } } }
I20260925 12:17:29.656569 5784 sys_catalog.cc:458] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:29.656533 5785 sys_catalog.cc:455] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d08378e8dead4c2a917ae0f0ce652e62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d08378e8dead4c2a917ae0f0ce652e62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37155 } } }
I20260925 12:17:29.656599 5785 sys_catalog.cc:458] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:29.656744 5789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:29.657495 5789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:17:29.659883 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:29.653420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50122 (local address 127.0.98.62:37155)
0925 12:17:29.653616 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:29.653624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:29.653659 (+ 35us) server_negotiation.cc:407] Connection header received
0925 12:17:29.654798 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:29.654803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:29.654853 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:29.654987 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:29.655222 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.656750 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.657744 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.658318 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.658469 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:29.659120 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:29.659124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:29.659124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:29.659322 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:29.659359 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:29.659373 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:29.659474 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:29.659590 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 12:17:29.659686 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260925 12:17:29.660383 5789 catalog_manager.cc:1347] Generated new cluster ID: f27dfe72015f4ec8bb9cbbee449e596e
I20260925 12:17:29.660435 5789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:17:29.664142 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:17:29.683377 5789 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:17:29.684069 5789 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:29.692070 5789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d08378e8dead4c2a917ae0f0ce652e62: Generated new TSK 0
I20260925 12:17:29.692222 5789 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:17:29.700860 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:29.697637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50130 (local address 127.0.98.62:37155)
0925 12:17:29.697687 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:29.697689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:29.697760 (+ 71us) server_negotiation.cc:407] Connection header received
0925 12:17:29.697794 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:29.697794 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:29.697805 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:29.697880 (+ 75us) server_negotiation.cc:226] Negotiated authn=JWT
0925 12:17:29.697989 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.698679 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.699344 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.699668 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.699748 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:29.699876 (+ 128us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 12:17:29.700688 (+ 812us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43511/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 12:17:29.700799 (+ 111us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50130: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43511/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":31}
W20260925 12:17:29.700925 5783 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50130: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43511/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 12:17:29.702790 5808 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:29.697568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37155 (local address 127.0.0.1:50130)
0925 12:17:29.697704 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:29.697712 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:29.697756 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:29.697895 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:29.697895 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:29.697897 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 12:17:29.697935 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:29.697936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.698754 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.698754 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:29.699275 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:29.699276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.699752 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.699758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:29.699794 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:29.699798 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 12:17:29.700777 (+ 979us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43511/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 12:17:29.702693 (+ 1916us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37155: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43511/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":1740,"thread_start_us":87,"threads_started":1}
I20260925 12:17:29.703176 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 5712
E20260925 12:17:29.712954 5809 webserver.cc:533] Webserver: sslize failed:
E20260925 12:17:29.713007 5809 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T12:17:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (210 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T12:17:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:29Z Disabled control of system clock
I20260925 12:17:29.747991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34637
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34637
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338625091267-392-0/testchainca.cert with env {}
W20260925 12:17:29.821046 5813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:29.821221 5813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:29.821241 5813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:29.822381 5813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:29.822422 5813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:29.822434 5813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:29.822448 5813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:29.822501 5813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 12:17:29.822523 5813 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 12:17:29.822548 5813 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 12:17:29.823992 5813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34637
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338625091267-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:29.824242 5813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:29.824447 5813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:29.829075 5821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:29.829075 5818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:29.829098 5819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:29.829267 5813 server_base.cc:1034] running on GCE node
I20260925 12:17:29.829505 5813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:29.829725 5813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:29.830857 5813 hybrid_clock.cc:648] HybridClock initialized: now 1790338649830846 us; error 46 us; skew 500 ppm
I20260925 12:17:29.832067 5813 webserver.cc:460] Webserver started at http://127.0.98.62:44221/ using document root <none> and password file <none>
I20260925 12:17:29.832291 5813 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:29.832334 5813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:29.832482 5813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:29.833479 5813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "5bc004a6aff54c498e55ac62f407c649"
format_stamp: "Formatted at 2026-09-25 12:17:29 on dist-test-slave-clmx"
server_key: "ca58ced612c89893ab16660af4d65ec0"
server_key_iv: "e2f07ed8d1d8bfe0683787feb9ab8c45"
server_key_version: "encryptionkey@0"
I20260925 12:17:29.833839 5813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "5bc004a6aff54c498e55ac62f407c649"
format_stamp: "Formatted at 2026-09-25 12:17:29 on dist-test-slave-clmx"
server_key: "ca58ced612c89893ab16660af4d65ec0"
server_key_iv: "e2f07ed8d1d8bfe0683787feb9ab8c45"
server_key_version: "encryptionkey@0"
I20260925 12:17:29.834976 5813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 12:17:29.835590 5827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:29.835775 5813 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:29.835866 5813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "5bc004a6aff54c498e55ac62f407c649"
format_stamp: "Formatted at 2026-09-25 12:17:29 on dist-test-slave-clmx"
server_key: "ca58ced612c89893ab16660af4d65ec0"
server_key_iv: "e2f07ed8d1d8bfe0683787feb9ab8c45"
server_key_version: "encryptionkey@0"
I20260925 12:17:29.835944 5813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:29.866400 5813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:29.866665 5813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:29.866809 5813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:29.870272 5813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34637
I20260925 12:17:29.870304 5879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34637 every 8 connection(s)
I20260925 12:17:29.870611 5813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:29.871481 5880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:29.871757 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 5813
I20260925 12:17:29.871860 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:17:29.872041 392 external_mini_cluster.cc:1428] Setting key e072e4fc38e2b2b9813c4c20defc74ea
I20260925 12:17:29.874084 5880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649: Bootstrap starting.
I20260925 12:17:29.874758 5880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:29.875041 5880 log.cc:824] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:29.875720 5880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649: No bootstrap required, opened a new log
I20260925 12:17:29.877202 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:29.872525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51192 (local address 127.0.98.62:34637)
0925 12:17:29.872715 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:29.872721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:29.872744 (+ 23us) server_negotiation.cc:407] Connection header received
0925 12:17:29.873133 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:29.873139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:29.873192 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:29.873321 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:29.873550 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.874388 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.875475 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.875926 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.876009 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:29.876608 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:29.876611 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:29.876611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:29.876751 (+ 140us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:29.876778 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:29.876788 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:29.876835 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:29.876943 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 12:17:29.877012 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":88,"threads_started":1}
I20260925 12:17:29.878019 5880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc004a6aff54c498e55ac62f407c649" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } }
I20260925 12:17:29.878166 5880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:29.878180 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:17:29.878211 5880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc004a6aff54c498e55ac62f407c649, State: Initialized, Role: FOLLOWER
I20260925 12:17:29.878315 5880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc004a6aff54c498e55ac62f407c649" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } }
I20260925 12:17:29.878398 5880 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:29.878435 5880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:29.878477 5880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:29.878980 5880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc004a6aff54c498e55ac62f407c649" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } }
I20260925 12:17:29.879072 5880 leader_election.cc:302] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bc004a6aff54c498e55ac62f407c649; no voters:
I20260925 12:17:29.879199 5880 leader_election.cc:288] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:17:29.879223 5885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:29.879315 5885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649 [term 1 LEADER]: Becoming Leader. State: Replica: 5bc004a6aff54c498e55ac62f407c649, State: Running, Role: LEADER
I20260925 12:17:29.879384 5885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc004a6aff54c498e55ac62f407c649" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } }
I20260925 12:17:29.879412 5880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:29.879700 5887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bc004a6aff54c498e55ac62f407c649. Latest consensus state: current_term: 1 leader_uuid: "5bc004a6aff54c498e55ac62f407c649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc004a6aff54c498e55ac62f407c649" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } }
I20260925 12:17:29.879711 5886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bc004a6aff54c498e55ac62f407c649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc004a6aff54c498e55ac62f407c649" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } }
I20260925 12:17:29.879782 5887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:29.879801 5886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649 [sys.catalog]: This master's current role is: LEADER
W20260925 12:17:29.880821 5901 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:17:29.880877 5901 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:17:29.880925 5891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:29.881047 5891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:17:29.882288 5891 catalog_manager.cc:1347] Generated new cluster ID: 933a4c5b93774e31a9dd6a2528ffdd41
I20260925 12:17:29.882336 5891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:17:29.893795 5891 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:17:29.894433 5891 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:29.902285 5891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bc004a6aff54c498e55ac62f407c649: Generated new TSK 0
I20260925 12:17:29.902456 5891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:17:30.360390 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:29.909500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51194 (local address 127.0.98.62:34637)
0925 12:17:29.909541 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:29.909543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:29.909644 (+ 101us) server_negotiation.cc:407] Connection header received
0925 12:17:29.909708 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:29.909709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:29.909718 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:29.909771 (+ 53us) server_negotiation.cc:226] Negotiated authn=JWT
0925 12:17:29.909895 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.910320 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.911388 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.911553 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.911594 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:29.911803 (+ 209us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 12:17:30.360186 (+448383us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 12:17:30.360333 (+ 147us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51194: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 12:17:30.360442 5883 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51194: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 12:17:30.360550 5909 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:29.909411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.0.1:51194)
0925 12:17:29.909565 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:29.909575 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:29.909647 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:29.909784 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:29.909785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:29.909787 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 12:17:29.909837 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:29.909839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.910387 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.910388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:29.911300 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:29.911302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:29.911639 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:29.911645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:29.911701 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:29.911706 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 12:17:30.360319 (+448613us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 12:17:30.360416 (+ 97us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34637: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260925 12:17:30.360823 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 5813
2026-09-25T12:17:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (636 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (81 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (82 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:17:30 dist-test-slave-clmx krb5kdc[5921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:17:30 dist-test-slave-clmx krb5kdc[5921](info): set up 2 sockets
Sep 25 12:17:30 dist-test-slave-clmx krb5kdc[5921](info): commencing operation
krb5kdc: starting...
W20260925 12:17:31.523552 392 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:17:31 dist-test-slave-clmx krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:17:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:17:31.628057 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40827
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:31.702172 5934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:31.702361 5934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:31.702389 5934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:31.703516 5934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:31.703552 5934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:31.703584 5934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:31.703596 5934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:31.703660 5934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:17:31.705117 5934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:31.705294 5934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:31.705513 5934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:31.709862 5942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:31.709875 5939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:31.709944 5940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:31.710055 5934 server_base.cc:1034] running on GCE node
I20260925 12:17:31.710325 5934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:31.710580 5934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:31.711730 5934 hybrid_clock.cc:648] HybridClock initialized: now 1790338651711708 us; error 32 us; skew 500 ppm
Sep 25 12:17:31 dist-test-slave-clmx krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:31.714632 5934 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:17:31.715119 5934 webserver.cc:460] Webserver started at http://127.0.98.62:45295/ using document root <none> and password file <none>
I20260925 12:17:31.715409 5934 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:31.715473 5934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:31.715646 5934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:31.716645 5934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "1c289e3847e04a48aaa8de6034e083fe"
format_stamp: "Formatted at 2026-09-25 12:17:31 on dist-test-slave-clmx"
server_key: "ea94b0c862bb365ccff7bbb3977b6dab"
server_key_iv: "441051d711236f27162e921e48008c04"
server_key_version: "encryptionkey@0"
I20260925 12:17:31.716980 5934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "1c289e3847e04a48aaa8de6034e083fe"
format_stamp: "Formatted at 2026-09-25 12:17:31 on dist-test-slave-clmx"
server_key: "ea94b0c862bb365ccff7bbb3977b6dab"
server_key_iv: "441051d711236f27162e921e48008c04"
server_key_version: "encryptionkey@0"
I20260925 12:17:31.718181 5934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:17:31.718811 5949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:31.718973 5934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:31.719063 5934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "1c289e3847e04a48aaa8de6034e083fe"
format_stamp: "Formatted at 2026-09-25 12:17:31 on dist-test-slave-clmx"
server_key: "ea94b0c862bb365ccff7bbb3977b6dab"
server_key_iv: "441051d711236f27162e921e48008c04"
server_key_version: "encryptionkey@0"
I20260925 12:17:31.719141 5934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:31.736708 5934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:31.738754 5934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:31.738893 5934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:31.742712 5934 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40827
I20260925 12:17:31.742741 6001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40827 every 8 connection(s)
I20260925 12:17:31.743176 5934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:31.743803 6002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:31.745885 6002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe: Bootstrap starting.
I20260925 12:17:31.746414 6002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:31.746671 6002 log.cc:824] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:31.747259 6002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe: No bootstrap required, opened a new log
I20260925 12:17:31.748755 6002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c289e3847e04a48aaa8de6034e083fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40827 } }
I20260925 12:17:31.748869 6002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:31.748888 6002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c289e3847e04a48aaa8de6034e083fe, State: Initialized, Role: FOLLOWER
I20260925 12:17:31.749068 6002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c289e3847e04a48aaa8de6034e083fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40827 } }
I20260925 12:17:31.749133 6002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:31.749169 6002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:31.749213 6002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:31.749964 6002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c289e3847e04a48aaa8de6034e083fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40827 } }
I20260925 12:17:31.750062 6002 leader_election.cc:302] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c289e3847e04a48aaa8de6034e083fe; no voters:
I20260925 12:17:31.750217 6002 leader_election.cc:288] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:17:31.750247 6005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:31.750398 6005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe [term 1 LEADER]: Becoming Leader. State: Replica: 1c289e3847e04a48aaa8de6034e083fe, State: Running, Role: LEADER
I20260925 12:17:31.750428 6002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:31.750519 6005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c289e3847e04a48aaa8de6034e083fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40827 } }
I20260925 12:17:31.750809 6006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c289e3847e04a48aaa8de6034e083fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c289e3847e04a48aaa8de6034e083fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40827 } } }
I20260925 12:17:31.750818 6007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c289e3847e04a48aaa8de6034e083fe. Latest consensus state: current_term: 1 leader_uuid: "1c289e3847e04a48aaa8de6034e083fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c289e3847e04a48aaa8de6034e083fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40827 } } }
I20260925 12:17:31.750931 6006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:31.750982 6007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:31.751385 6014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:31.751749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 5934
I20260925 12:17:31.751858 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:17:31.751955 6014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:17:31.752053 392 external_mini_cluster.cc:1428] Setting key c0be9ae248911c76e5dd9199bd514781
I20260925 12:17:31.753350 6014 catalog_manager.cc:1347] Generated new cluster ID: 38dcb863be154eea8cb0f91cf20b9224
I20260925 12:17:31.753400 6014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:17:31 dist-test-slave-clmx krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:31.762663 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:31.752587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.0.98.62:40827)
0925 12:17:31.752739 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:31.752748 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:31.752778 (+ 30us) server_negotiation.cc:407] Connection header received
0925 12:17:31.752882 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:31.752889 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:31.752950 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:31.753076 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:31.753898 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:31.755206 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:31.756185 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:31.756726 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:31.756810 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:31.759229 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:31.759233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:31.759233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:31.759243 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:31.761141 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:31.761513 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:31.761516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:31.761516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:31.761616 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:31.761819 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:31.761820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:31.761820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:31.762095 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:31.762267 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:31.762338 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 12:17:31.762470 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":92,"threads_started":1}
I20260925 12:17:31.770498 6014 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:17:31.771127 6014 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:31.775972 6014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c289e3847e04a48aaa8de6034e083fe: Generated new TSK 0
I20260925 12:17:31.776106 6014 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:17:31.801879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40827
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:31.877731 6028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:31.877910 6028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:31.877930 6028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:31.878952 6028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:31.878989 6028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:31.879012 6028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:17:31.880331 6028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:31.880532 6028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:31.880743 6028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:31.885334 6033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:31.885293 6034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:31.885308 6036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:31.885521 6028 server_base.cc:1034] running on GCE node
I20260925 12:17:31.885715 6028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:31.885947 6028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:31.887097 6028 hybrid_clock.cc:648] HybridClock initialized: now 1790338651887089 us; error 37 us; skew 500 ppm
Sep 25 12:17:31 dist-test-slave-clmx krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:31.889465 6028 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:17:31.889858 6028 webserver.cc:460] Webserver started at http://127.0.98.1:39077/ using document root <none> and password file <none>
I20260925 12:17:31.890017 6028 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:31.890053 6028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:31.890127 6028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:31.890993 6028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "00d1730516784dc98ba3c4068d470259"
format_stamp: "Formatted at 2026-09-25 12:17:31 on dist-test-slave-clmx"
server_key: "e87b4a79bf20ebb31c01bf50c41d6467"
server_key_iv: "1702f4755e1b193451a476a10e884a3b"
server_key_version: "encryptionkey@0"
I20260925 12:17:31.891290 6028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "00d1730516784dc98ba3c4068d470259"
format_stamp: "Formatted at 2026-09-25 12:17:31 on dist-test-slave-clmx"
server_key: "e87b4a79bf20ebb31c01bf50c41d6467"
server_key_iv: "1702f4755e1b193451a476a10e884a3b"
server_key_version: "encryptionkey@0"
I20260925 12:17:31.892521 6028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:17:31.893127 6043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:31.893272 6028 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:17:31.893347 6028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "00d1730516784dc98ba3c4068d470259"
format_stamp: "Formatted at 2026-09-25 12:17:31 on dist-test-slave-clmx"
server_key: "e87b4a79bf20ebb31c01bf50c41d6467"
server_key_iv: "1702f4755e1b193451a476a10e884a3b"
server_key_version: "encryptionkey@0"
I20260925 12:17:31.893404 6028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:31.914265 6028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:31.915964 6028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:31.916064 6028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:31.916226 6028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:31.916504 6028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:31.916535 6028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:31.916556 6028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:31.916599 6028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:31.921867 6028 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43985
I20260925 12:17:31.921928 6156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43985 every 8 connection(s)
I20260925 12:17:31.922205 6028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:17:31.925940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 6028
I20260925 12:17:31.926045 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:17:31.926210 392 external_mini_cluster.cc:1428] Setting key c2516053950ac199362b957aee374e4d
Sep 25 12:17:31 dist-test-slave-clmx krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:31.931756 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:31.922948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41195 (local address 127.0.98.62:40827)
0925 12:17:31.922995 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:31.922997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:31.923216 (+ 219us) server_negotiation.cc:407] Connection header received
0925 12:17:31.924184 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:31.924185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:31.924199 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:31.924246 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:31.924787 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:31.925250 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:31.926487 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:31.926684 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:31.926769 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:31.929033 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:31.929035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:31.929035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:31.929041 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:31.930418 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:31.930827 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:31.930828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:31.930828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:31.930883 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:31.931126 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:31.931127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:31.931127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:31.931235 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:31.931309 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:31.931568 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 12:17:31.931666 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:17:31.931926 6159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:31.922948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40827 (local address 127.0.98.1:41195)
0925 12:17:31.923105 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:31.923137 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:31.924100 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:31.924303 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:31.924306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:31.924542 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:31.924724 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:31.924727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:31.925322 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:31.925322 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:31.926394 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:31.926396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:31.926776 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:31.926777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:31.926935 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:31.927401 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:31.927411 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:31.928929 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:31.930508 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:31.930515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:31.930517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:31.930745 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:31.930954 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:31.930955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:31.930955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:31.931061 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:31.931301 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:31.931307 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:31.931444 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 12:17:31.931547 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 12:17:31.931709 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260925 12:17:31.932492 6157 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40827
I20260925 12:17:31.932588 6157 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:31.932771 6157 heartbeater.cc:511] Master 127.0.98.62:40827 requested a full tablet report, sending...
I20260925 12:17:31.933236 5966 ts_manager.cc:194] Registered new tserver with Master: 00d1730516784dc98ba3c4068d470259 (127.0.98.1:43985)
I20260925 12:17:31.933816 5966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41195
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:17:31.948124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40827
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:32.021781 6162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:32.021994 6162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:32.022017 6162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:32.023243 6162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:32.023283 6162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:32.023329 6162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:17:32.024785 6162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:32.024996 6162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:32.025187 6162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:32.029541 6167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:32.029548 6170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:32.029554 6168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:32.029744 6162 server_base.cc:1034] running on GCE node
I20260925 12:17:32.030174 6162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:32.030392 6162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:32.031538 6162 hybrid_clock.cc:648] HybridClock initialized: now 1790338652031526 us; error 23 us; skew 500 ppm
Sep 25 12:17:32 dist-test-slave-clmx krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:32.033778 6162 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:17:32.034161 6162 webserver.cc:460] Webserver started at http://127.0.98.2:43307/ using document root <none> and password file <none>
I20260925 12:17:32.034355 6162 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:32.034420 6162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:32.034570 6162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:32.035593 6162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "14d5d9cb73134f1eafe1e404a094a603"
format_stamp: "Formatted at 2026-09-25 12:17:32 on dist-test-slave-clmx"
server_key: "745c59f16ed425d719ea775036e6c60e"
server_key_iv: "aeef3d5c0090649e90add5bf4a9b40f3"
server_key_version: "encryptionkey@0"
I20260925 12:17:32.035984 6162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "14d5d9cb73134f1eafe1e404a094a603"
format_stamp: "Formatted at 2026-09-25 12:17:32 on dist-test-slave-clmx"
server_key: "745c59f16ed425d719ea775036e6c60e"
server_key_iv: "aeef3d5c0090649e90add5bf4a9b40f3"
server_key_version: "encryptionkey@0"
I20260925 12:17:32.037099 6162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:32.037760 6177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:32.037914 6162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:32.038004 6162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "14d5d9cb73134f1eafe1e404a094a603"
format_stamp: "Formatted at 2026-09-25 12:17:32 on dist-test-slave-clmx"
server_key: "745c59f16ed425d719ea775036e6c60e"
server_key_iv: "aeef3d5c0090649e90add5bf4a9b40f3"
server_key_version: "encryptionkey@0"
I20260925 12:17:32.038082 6162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:32.044894 6162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:32.046437 6162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:32.046581 6162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:32.046772 6162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:32.047040 6162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:32.047094 6162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:32.047133 6162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:32.047158 6162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:32.052886 6162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44313
I20260925 12:17:32.052922 6290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44313 every 8 connection(s)
I20260925 12:17:32.053203 6162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:17:32 dist-test-slave-clmx krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:32.060854 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:32.053896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47181 (local address 127.0.98.62:40827)
0925 12:17:32.053961 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:32.053963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:32.054166 (+ 203us) server_negotiation.cc:407] Connection header received
0925 12:17:32.054931 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:32.054932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:32.054941 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:32.054986 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:32.055452 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:32.055938 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:32.056746 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:32.056902 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:32.056945 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:32.058648 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:32.058649 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:32.058649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:32.058655 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:32.059826 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:32.060168 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:32.060169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:32.060169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:32.060213 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:32.060411 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:32.060412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:32.060412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:32.060498 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:32.060543 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:32.060738 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 12:17:32.060797 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 12:17:32.061007 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:32.053852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40827 (local address 127.0.98.2:47181)
0925 12:17:32.054041 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:32.054070 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:32.054849 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:32.055029 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:32.055032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:32.055233 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:32.055391 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:32.055394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:32.056002 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:32.056002 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:32.056680 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:32.056682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:32.056963 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:32.056964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:32.057045 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:32.057386 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:32.057393 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:32.058562 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:32.059899 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:32.059904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:32.059905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:32.060109 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:32.060265 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:32.060266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:32.060266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:32.060349 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:32.060548 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:32.060551 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:32.060642 (+ 91us) client_negotiation.cc:769] Sending connection context
0925 12:17:32.060715 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 12:17:32.060813 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":109,"threads_started":1}
I20260925 12:17:32.061542 6291 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40827
I20260925 12:17:32.061623 6291 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:32.061766 6291 heartbeater.cc:511] Master 127.0.98.62:40827 requested a full tablet report, sending...
I20260925 12:17:32.062089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 6162
I20260925 12:17:32.062090 5966 ts_manager.cc:194] Registered new tserver with Master: 14d5d9cb73134f1eafe1e404a094a603 (127.0.98.2:44313)
I20260925 12:17:32.062161 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:17:32.062382 392 external_mini_cluster.cc:1428] Setting key 5e7673db44fe0ffd33c05d7a1cccec24
I20260925 12:17:32.062558 5966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47181
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:17:32.083812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40827
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:32.158830 6296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:32.159035 6296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:32.159071 6296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:32.160264 6296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:32.160327 6296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:32.160372 6296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:17:32.161800 6296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:32.162032 6296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:32.162295 6296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:32.166893 6304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:32.166926 6301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:32.166911 6302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:32.167021 6296 server_base.cc:1034] running on GCE node
I20260925 12:17:32.167251 6296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:32.167498 6296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:32.168627 6296 hybrid_clock.cc:648] HybridClock initialized: now 1790338652168614 us; error 31 us; skew 500 ppm
Sep 25 12:17:32 dist-test-slave-clmx krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:32.170802 6296 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:17:32.171180 6296 webserver.cc:460] Webserver started at http://127.0.98.3:41079/ using document root <none> and password file <none>
I20260925 12:17:32.171388 6296 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:32.171458 6296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:32.171591 6296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:32.172559 6296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "458aaacc9e974e08b137eb5d93323cab"
format_stamp: "Formatted at 2026-09-25 12:17:32 on dist-test-slave-clmx"
server_key: "ee6a2a4f111cb3ce356063c9fa5b0155"
server_key_iv: "5f609cc0fa1513d7d521b7eb9e2c7b10"
server_key_version: "encryptionkey@0"
I20260925 12:17:32.172895 6296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "458aaacc9e974e08b137eb5d93323cab"
format_stamp: "Formatted at 2026-09-25 12:17:32 on dist-test-slave-clmx"
server_key: "ee6a2a4f111cb3ce356063c9fa5b0155"
server_key_iv: "5f609cc0fa1513d7d521b7eb9e2c7b10"
server_key_version: "encryptionkey@0"
I20260925 12:17:32.174111 6296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:17:32.174757 6311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:32.174945 6296 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:17:32.175036 6296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "458aaacc9e974e08b137eb5d93323cab"
format_stamp: "Formatted at 2026-09-25 12:17:32 on dist-test-slave-clmx"
server_key: "ee6a2a4f111cb3ce356063c9fa5b0155"
server_key_iv: "5f609cc0fa1513d7d521b7eb9e2c7b10"
server_key_version: "encryptionkey@0"
I20260925 12:17:32.175155 6296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:32.192009 6296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:32.193640 6296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:32.193785 6296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:32.193977 6296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:32.194250 6296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:32.194304 6296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:32.194342 6296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:32.194368 6296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:32.199718 6296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37079
I20260925 12:17:32.199784 6424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37079 every 8 connection(s)
I20260925 12:17:32.200074 6296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 12:17:32 dist-test-slave-clmx krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:32.207561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 6296
I20260925 12:17:32.207690 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:17:32.207893 392 external_mini_cluster.cc:1428] Setting key c44000653b3699e41f4a49e3d0712b7f
I20260925 12:17:32.208807 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:32.200709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53973 (local address 127.0.98.62:40827)
0925 12:17:32.200749 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:32.200751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:32.200935 (+ 184us) server_negotiation.cc:407] Connection header received
0925 12:17:32.201811 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:32.201813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:32.201825 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:32.201874 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:32.202453 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:32.202951 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:32.203977 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:32.204195 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:32.204264 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:32.206047 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:32.206048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:32.206048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:32.206055 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:32.207199 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:32.207770 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:32.207773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:32.207773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:32.207837 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:32.208105 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:32.208107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:32.208107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:32.208258 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:32.208324 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:32.208630 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 12:17:32.208712 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:17:32.208962 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:32.200675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40827 (local address 127.0.98.3:53973)
0925 12:17:32.200831 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:32.200860 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:32.201700 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:32.201930 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:32.201934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:32.202163 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:32.202358 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:32.202362 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:32.203039 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:32.203040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:32.203882 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:32.203884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:32.204281 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:32.204282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:32.204391 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:32.204760 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:32.204767 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:32.205956 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:32.207294 (+ 1338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:32.207299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:32.207301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:32.207694 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:32.207900 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:32.207901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:32.207901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:32.208032 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:32.208358 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:32.208362 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:32.208526 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 12:17:32.208606 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 12:17:32.208742 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260925 12:17:32.209328 6425 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40827
I20260925 12:17:32.209425 6425 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:32.209576 6425 heartbeater.cc:511] Master 127.0.98.62:40827 requested a full tablet report, sending...
I20260925 12:17:32.209914 5966 ts_manager.cc:194] Registered new tserver with Master: 458aaacc9e974e08b137eb5d93323cab (127.0.98.3:37079)
I20260925 12:17:32.210259 5966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53973
I20260925 12:17:32.219539 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/test-admin.keytab.
Sep 25 12:17:32 dist-test-slave-clmx krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:32.233486 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 12:17:32.234858 6432 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:32.234161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53354 (local address 127.0.98.1:43985)
0925 12:17:32.234301 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:32.234305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:32.234315 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:17:32.234344 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:32.234345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:32.234357 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:32.234427 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:32.234781 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53354: BlockingRecv error: recv got EOF from 127.0.0.1:53354 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
W20260925 12:17:32.234939 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 12:17:32 dist-test-slave-clmx krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:32.235893 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 12:17:32 dist-test-slave-clmx krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:32.236727 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 12:17:32 dist-test-slave-clmx krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:17:32.243942 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:32.237175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53366 (local address 127.0.98.1:43985)
0925 12:17:32.237242 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:32.237243 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:32.237283 (+ 40us) server_negotiation.cc:407] Connection header received
0925 12:17:32.237316 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:32.237317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:32.237325 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:32.237366 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:32.237666 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:32.238342 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:32.238991 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:32.239255 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:32.239303 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:32.241343 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:32.241349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:32.241349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:32.241355 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:32.242954 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:32.243238 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:32.243240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:32.243240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:32.243286 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:32.243432 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:32.243432 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:32.243433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:32.243540 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:32.243669 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:32.243743 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 12:17:32.243828 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 12:17:32.245427 6432 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:32.244953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53380 (local address 127.0.98.1:43985)
0925 12:17:32.244983 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:32.244985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:32.245058 (+ 73us) server_negotiation.cc:407] Connection header received
0925 12:17:32.245105 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:32.245106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:32.245114 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:32.245147 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:32.245378 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53380: BlockingRecv error: recv got EOF from 127.0.0.1:53380 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 12:17:32.245471 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 12:17:32 dist-test-slave-clmx krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:32.246305 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 12:17:32 dist-test-slave-clmx krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:32.247059 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 12:17:32 dist-test-slave-clmx krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:17:32.253294 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:32.247518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53384 (local address 127.0.98.1:43985)
0925 12:17:32.247568 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:32.247570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:32.247630 (+ 60us) server_negotiation.cc:407] Connection header received
0925 12:17:32.247695 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:32.247695 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:32.247703 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:32.247759 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:32.248111 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:32.248526 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:32.249138 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:32.249308 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:32.249388 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:32.251092 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:32.251093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:32.251093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:32.251099 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:32.252269 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:32.252607 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:32.252608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:32.252608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:32.252657 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:32.252865 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:32.252865 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:32.252866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:32.252957 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:32.253050 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:32.253145 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 12:17:32.253213 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260925 12:17:32.254489 6432 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:32.254115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53398 (local address 127.0.98.1:43985)
0925 12:17:32.254142 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:32.254143 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:32.254184 (+ 41us) server_negotiation.cc:407] Connection header received
0925 12:17:32.254249 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:32.254250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:32.254256 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:32.254301 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:32.254468 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53398: BlockingRecv error: recv got EOF from 127.0.0.1:53398 (error 108)
Metrics: {"server-negotiator.queue_time_us":13}
W20260925 12:17:32.254684 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338625091267-392-0/krb5kdc/krb5cc)
Sep 25 12:17:32 dist-test-slave-clmx krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:32.255445 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 12:17:32 dist-test-slave-clmx krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:32.256138 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 12:17:32 dist-test-slave-clmx krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:17:32.261734 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:32.256515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53400 (local address 127.0.98.1:43985)
0925 12:17:32.256568 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:32.256569 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:32.256575 (+ 6us) server_negotiation.cc:407] Connection header received
0925 12:17:32.256629 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:32.256629 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:32.256635 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:32.256665 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:32.256944 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:32.257332 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:32.257907 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:32.258053 (+ 146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:32.258095 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:32.259924 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:32.259925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:32.259925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:32.259930 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:32.260950 (+ 1020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:32.261235 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:32.261236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:32.261236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:32.261280 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:32.261417 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:32.261417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:32.261417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:32.261476 (+ 59us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:32.261526 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:32.261585 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 12:17:32.261626 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 12:17:32.262280 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 6028
I20260925 12:17:32.266276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 6162
I20260925 12:17:32.269948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 6296
I20260925 12:17:32.273917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 5934
2026-09-25T12:17:32Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1750 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:17:32 dist-test-slave-clmx krb5kdc[6445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:17:32 dist-test-slave-clmx krb5kdc[6445](info): set up 2 sockets
Sep 25 12:17:32 dist-test-slave-clmx krb5kdc[6445](info): commencing operation
krb5kdc: starting...
W20260925 12:17:33.267342 392 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:17:33 dist-test-slave-clmx krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:17:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260925 12:17:33.369992 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40821
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:33.441888 6458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:33.442076 6458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:33.442108 6458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:33.443228 6458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:33.443295 6458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:33.443320 6458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:33.443346 6458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:33.443373 6458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:17:33.444814 6458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:33.445080 6458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:33.445307 6458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:33.449620 6463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:33.449633 6464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:33.449796 6458 server_base.cc:1034] running on GCE node
W20260925 12:17:33.449633 6466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:33.450057 6458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:33.450312 6458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:33.451483 6458 hybrid_clock.cc:648] HybridClock initialized: now 1790338653451479 us; error 40 us; skew 500 ppm
Sep 25 12:17:33 dist-test-slave-clmx krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338653, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:33.460551 6458 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260925 12:17:33.460989 6458 webserver.cc:460] Webserver started at http://127.0.98.62:34883/ using document root <none> and password file <none>
I20260925 12:17:33.461251 6458 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:33.461314 6458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:33.461436 6458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:33.462425 6458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "8b159ed78d924726b4d7a669d916a880"
format_stamp: "Formatted at 2026-09-25 12:17:33 on dist-test-slave-clmx"
server_key: "fc7ffd336b56049798fb5e9dcf2da4a7"
server_key_iv: "6466dfc29e46bf2140ebf459499b0dec"
server_key_version: "encryptionkey@0"
I20260925 12:17:33.462764 6458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "8b159ed78d924726b4d7a669d916a880"
format_stamp: "Formatted at 2026-09-25 12:17:33 on dist-test-slave-clmx"
server_key: "fc7ffd336b56049798fb5e9dcf2da4a7"
server_key_iv: "6466dfc29e46bf2140ebf459499b0dec"
server_key_version: "encryptionkey@0"
I20260925 12:17:33.464612 6458 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 12:17:33.465610 6473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:33.465796 6458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:33.465972 6458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "8b159ed78d924726b4d7a669d916a880"
format_stamp: "Formatted at 2026-09-25 12:17:33 on dist-test-slave-clmx"
server_key: "fc7ffd336b56049798fb5e9dcf2da4a7"
server_key_iv: "6466dfc29e46bf2140ebf459499b0dec"
server_key_version: "encryptionkey@0"
I20260925 12:17:33.466058 6458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:33.479919 6458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:33.481505 6458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:33.481655 6458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:33.485327 6458 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40821
I20260925 12:17:33.485388 6525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40821 every 8 connection(s)
I20260925 12:17:33.485699 6458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:33.486586 6526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:33.488758 6526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880: Bootstrap starting.
I20260925 12:17:33.489295 6526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:33.489562 6526 log.cc:824] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:33.490199 6526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880: No bootstrap required, opened a new log
I20260925 12:17:33.491730 6526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } }
I20260925 12:17:33.491850 6526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:33.491871 6526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b159ed78d924726b4d7a669d916a880, State: Initialized, Role: FOLLOWER
I20260925 12:17:33.492070 6526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } }
I20260925 12:17:33.492161 6526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:33.492218 6526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:33.492264 6526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:33.493036 6526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } }
I20260925 12:17:33.493144 6526 leader_election.cc:302] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b159ed78d924726b4d7a669d916a880; no voters:
I20260925 12:17:33.493310 6526 leader_election.cc:288] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:17:33.493331 6529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:33.493672 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 6458
I20260925 12:17:33.493773 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:17:33.493921 392 external_mini_cluster.cc:1428] Setting key d655d719417c2ebdb2d174b7e5078e8d
I20260925 12:17:33.493486 6529 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [term 1 LEADER]: Becoming Leader. State: Replica: 8b159ed78d924726b4d7a669d916a880, State: Running, Role: LEADER
I20260925 12:17:33.493554 6526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:33.494174 6529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } }
I20260925 12:17:33.494879 6531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b159ed78d924726b4d7a669d916a880. Latest consensus state: current_term: 1 leader_uuid: "8b159ed78d924726b4d7a669d916a880" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } }
I20260925 12:17:33.494863 6530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b159ed78d924726b4d7a669d916a880" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } }
I20260925 12:17:33.494963 6531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:33.494967 6530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:33.495223 6541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:33.495818 6541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:17:33.497290 6541 catalog_manager.cc:1347] Generated new cluster ID: 9ebbedcdd3ba4f3ca9c70ee49c9b30d5
I20260925 12:17:33.497340 6541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:17:33 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:17:33.504482 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:33.494490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47318 (local address 127.0.98.62:40821)
0925 12:17:33.494689 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:33.494700 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:33.494736 (+ 36us) server_negotiation.cc:407] Connection header received
0925 12:17:33.494859 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:33.494866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:33.494916 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:33.496015 (+ 1099us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:33.496120 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:33.497045 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:33.498064 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:33.498434 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:33.498506 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:33.500844 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:33.500848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:33.500848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:33.500859 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:33.502678 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:33.503032 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:33.503035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:33.503035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:33.503142 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:33.503311 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:33.503312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:33.503312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:33.503582 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:33.503746 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:33.504182 (+ 436us) server_negotiation.cc:299] Negotiation successful
0925 12:17:33.504298 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
I20260925 12:17:33.510200 6541 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:17:33.510838 6541 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:33.521318 6541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880: Generated new TSK 0
I20260925 12:17:33.521505 6541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260925 12:17:33.543861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40821
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:33.619356 6552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:33.619558 6552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:33.619594 6552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:33.620676 6552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:33.620736 6552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:33.620796 6552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:17:33.622186 6552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:33.622471 6552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:33.622694 6552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:33.627192 6560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:33.627166 6558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:33.627347 6552 server_base.cc:1034] running on GCE node
W20260925 12:17:33.627437 6557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:33.627678 6552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:33.627907 6552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:33.629038 6552 hybrid_clock.cc:648] HybridClock initialized: now 1790338653629028 us; error 26 us; skew 500 ppm
Sep 25 12:17:33 dist-test-slave-clmx krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338653, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:33.631197 6552 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 12:17:33.631662 6552 webserver.cc:460] Webserver started at http://127.0.98.1:42967/ using document root <none> and password file <none>
I20260925 12:17:33.631845 6552 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:33.631891 6552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:33.631970 6552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:33.632838 6552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "80fa57d079294d4589eb3e970a421c77"
format_stamp: "Formatted at 2026-09-25 12:17:33 on dist-test-slave-clmx"
server_key: "ae3dcc6ed50cd101c552cfb25a698dbe"
server_key_iv: "028e080823e96ef9adfda5712b20d367"
server_key_version: "encryptionkey@0"
I20260925 12:17:33.633126 6552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "80fa57d079294d4589eb3e970a421c77"
format_stamp: "Formatted at 2026-09-25 12:17:33 on dist-test-slave-clmx"
server_key: "ae3dcc6ed50cd101c552cfb25a698dbe"
server_key_iv: "028e080823e96ef9adfda5712b20d367"
server_key_version: "encryptionkey@0"
I20260925 12:17:33.634241 6552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:17:33.634847 6567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:33.635016 6552 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:33.635105 6552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "80fa57d079294d4589eb3e970a421c77"
format_stamp: "Formatted at 2026-09-25 12:17:33 on dist-test-slave-clmx"
server_key: "ae3dcc6ed50cd101c552cfb25a698dbe"
server_key_iv: "028e080823e96ef9adfda5712b20d367"
server_key_version: "encryptionkey@0"
I20260925 12:17:33.635181 6552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:33.643553 6552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:33.645192 6552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:33.645341 6552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:33.645537 6552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:33.645825 6552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:33.645876 6552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:33.645913 6552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:33.645941 6552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:33.651198 6552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38055
I20260925 12:17:33.651248 6680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38055 every 8 connection(s)
I20260925 12:17:33.651505 6552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 12:17:33 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338653, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:17:33.657747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 6552
I20260925 12:17:33.657862 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:17:33.658054 392 external_mini_cluster.cc:1428] Setting key 8417e644ff26fb2bef78e5987043a794
I20260925 12:17:33.659577 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:33.652090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.1:40309)
0925 12:17:33.652227 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:33.652261 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:33.652676 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:33.652865 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:33.652868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:33.653375 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:33.653541 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:33.653544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:33.654151 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:33.654152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:33.654908 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:33.654910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:33.655218 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:33.655219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:33.655309 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:33.655704 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:33.655711 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:33.656906 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:33.658358 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:33.658363 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:33.658365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:33.658574 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:33.658784 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:33.658785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:33.658785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:33.658888 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:33.659159 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:33.659163 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:33.659252 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 12:17:33.659339 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 12:17:33.659446 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260925 12:17:33.659605 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:33.652124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40309 (local address 127.0.98.62:40821)
0925 12:17:33.652179 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:33.652181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:33.652338 (+ 157us) server_negotiation.cc:407] Connection header received
0925 12:17:33.652751 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:33.652752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:33.652763 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:33.652820 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:33.653626 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:33.654079 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:33.654977 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:33.655154 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:33.655200 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:33.656976 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:33.656977 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:33.656977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:33.656982 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:33.658223 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:33.658668 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:33.658669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:33.658669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:33.658727 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:33.658952 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:33.658953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:33.658954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:33.659085 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:33.659164 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:33.659390 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 12:17:33.659474 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 12:17:33.660275 6681 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40821
I20260925 12:17:33.660370 6681 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:33.660534 6681 heartbeater.cc:511] Master 127.0.98.62:40821 requested a full tablet report, sending...
I20260925 12:17:33.660976 6490 ts_manager.cc:194] Registered new tserver with Master: 80fa57d079294d4589eb3e970a421c77 (127.0.98.1:38055)
I20260925 12:17:33.661579 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40309
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260925 12:17:33.679972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40821
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:33.755000 6686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:33.755162 6686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:33.755183 6686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:33.756335 6686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:33.756372 6686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:33.756417 6686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:17:33.757699 6686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:33.757910 6686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:33.758104 6686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:33.762560 6694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:33.762583 6686 server_base.cc:1034] running on GCE node
W20260925 12:17:33.762568 6692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:33.762590 6691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:33.762951 6686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:33.763221 6686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:33.764374 6686 hybrid_clock.cc:648] HybridClock initialized: now 1790338653764354 us; error 37 us; skew 500 ppm
Sep 25 12:17:33 dist-test-slave-clmx krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338653, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:33.766567 6686 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 12:17:33.766963 6686 webserver.cc:460] Webserver started at http://127.0.98.2:44317/ using document root <none> and password file <none>
I20260925 12:17:33.767162 6686 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:33.767243 6686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:33.767400 6686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:33.768365 6686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "a05341f431ea4d129ca4e5d7afdbf1d2"
format_stamp: "Formatted at 2026-09-25 12:17:33 on dist-test-slave-clmx"
server_key: "713626eb73db78ed79c606b93e16de4c"
server_key_iv: "31f67222a9157c5b637db4b05c50a2ad"
server_key_version: "encryptionkey@0"
I20260925 12:17:33.768692 6686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a05341f431ea4d129ca4e5d7afdbf1d2"
format_stamp: "Formatted at 2026-09-25 12:17:33 on dist-test-slave-clmx"
server_key: "713626eb73db78ed79c606b93e16de4c"
server_key_iv: "31f67222a9157c5b637db4b05c50a2ad"
server_key_version: "encryptionkey@0"
I20260925 12:17:33.769886 6686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:17:33.770519 6701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:33.770673 6686 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:17:33.770757 6686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "a05341f431ea4d129ca4e5d7afdbf1d2"
format_stamp: "Formatted at 2026-09-25 12:17:33 on dist-test-slave-clmx"
server_key: "713626eb73db78ed79c606b93e16de4c"
server_key_iv: "31f67222a9157c5b637db4b05c50a2ad"
server_key_version: "encryptionkey@0"
I20260925 12:17:33.770833 6686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:33.783384 6686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:33.785065 6686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:33.785163 6686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:33.785326 6686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:33.785564 6686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:33.785593 6686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:33.785611 6686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:33.785624 6686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:33.790706 6686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39153
I20260925 12:17:33.790755 6814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39153 every 8 connection(s)
I20260925 12:17:33.790992 6686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:17:33.793678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 6686
I20260925 12:17:33.793814 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:17:33.794046 392 external_mini_cluster.cc:1428] Setting key 5b1c0cc159f152c753ec2c93143cf466
Sep 25 12:17:33 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338653, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:17:33.800280 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:33.791673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.2:60823)
0925 12:17:33.791798 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:33.791829 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:33.792204 (+ 375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:33.792408 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:33.792410 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:33.792914 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:33.793098 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:33.793102 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:33.793742 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:33.793743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:33.794989 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:33.794992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:33.795428 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:33.795429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:33.795555 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:33.795982 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:33.795992 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:33.797445 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:33.799096 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:33.799101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:33.799102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:33.799311 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:33.799516 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:33.799516 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:33.799516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:33.799596 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:33.799901 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:33.799905 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:33.799997 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 12:17:33.800059 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 12:17:33.800153 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260925 12:17:33.800288 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:33.791673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60823 (local address 127.0.98.62:40821)
0925 12:17:33.791717 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:33.791719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:33.791891 (+ 172us) server_negotiation.cc:407] Connection header received
0925 12:17:33.792279 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:33.792280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:33.792290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:33.792341 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:33.793168 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:33.793662 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:33.795065 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:33.795353 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:33.795409 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:33.797527 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:33.797528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:33.797529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:33.797535 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:33.799023 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:33.799395 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:33.799396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:33.799396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:33.799452 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:33.799700 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:33.799701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:33.799701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:33.799832 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:33.799896 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:33.800117 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 12:17:33.800196 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 12:17:33.800675 6815 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40821
I20260925 12:17:33.800761 6815 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:33.800925 6815 heartbeater.cc:511] Master 127.0.98.62:40821 requested a full tablet report, sending...
I20260925 12:17:33.801231 6490 ts_manager.cc:194] Registered new tserver with Master: a05341f431ea4d129ca4e5d7afdbf1d2 (127.0.98.2:39153)
I20260925 12:17:33.801575 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60823
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 12:17:33.816434 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40821
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:33.891523 6820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:33.891706 6820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:33.891728 6820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:33.892874 6820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:33.892915 6820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:33.892966 6820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:17:33.894371 6820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:33.894563 6820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:33.894773 6820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:33.899057 6825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:33.899094 6826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:33.899153 6820 server_base.cc:1034] running on GCE node
W20260925 12:17:33.899101 6828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:33.899343 6820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:33.899562 6820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:33.900713 6820 hybrid_clock.cc:648] HybridClock initialized: now 1790338653900692 us; error 38 us; skew 500 ppm
Sep 25 12:17:33 dist-test-slave-clmx krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338653, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:33.902949 6820 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 12:17:33.903347 6820 webserver.cc:460] Webserver started at http://127.0.98.3:42427/ using document root <none> and password file <none>
I20260925 12:17:33.903508 6820 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:33.903558 6820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:33.903721 6820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:33.904598 6820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "ca1f60f9f9c040bdaf2f95beb2142646"
format_stamp: "Formatted at 2026-09-25 12:17:33 on dist-test-slave-clmx"
server_key: "7720f8f7b91dd88cf1b11d95ee2bfe5f"
server_key_iv: "6b45d9222f7f42e2e3a5c094237746a7"
server_key_version: "encryptionkey@0"
I20260925 12:17:33.904870 6820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ca1f60f9f9c040bdaf2f95beb2142646"
format_stamp: "Formatted at 2026-09-25 12:17:33 on dist-test-slave-clmx"
server_key: "7720f8f7b91dd88cf1b11d95ee2bfe5f"
server_key_iv: "6b45d9222f7f42e2e3a5c094237746a7"
server_key_version: "encryptionkey@0"
I20260925 12:17:33.906003 6820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:17:33.906574 6835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:33.906714 6820 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:33.906793 6820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "ca1f60f9f9c040bdaf2f95beb2142646"
format_stamp: "Formatted at 2026-09-25 12:17:33 on dist-test-slave-clmx"
server_key: "7720f8f7b91dd88cf1b11d95ee2bfe5f"
server_key_iv: "6b45d9222f7f42e2e3a5c094237746a7"
server_key_version: "encryptionkey@0"
I20260925 12:17:33.906843 6820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:33.920192 6820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:33.921711 6820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:33.921810 6820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:33.921957 6820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:33.922158 6820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:33.922186 6820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:33.922205 6820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:33.922217 6820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:33.927400 6820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37931
I20260925 12:17:33.927474 6948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37931 every 8 connection(s)
I20260925 12:17:33.927759 6820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:17:33.930282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 6820
I20260925 12:17:33.930363 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:17:33.930540 392 external_mini_cluster.cc:1428] Setting key 5d0ad2dd9337f2a6db9b37bfc401d475
Sep 25 12:17:33 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338653, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:17:33.936103 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:33.928408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48923 (local address 127.0.98.62:40821)
0925 12:17:33.928471 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:33.928472 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:33.928671 (+ 199us) server_negotiation.cc:407] Connection header received
0925 12:17:33.929065 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:33.929066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:33.929075 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:33.929129 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:33.929938 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:33.930548 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:33.931836 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:33.932012 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:33.932065 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:33.934012 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:33.934014 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:33.934014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:33.934020 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:33.935014 (+ 994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:33.935316 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:33.935316 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:33.935317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:33.935386 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:33.935574 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:33.935574 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:33.935575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:33.935722 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:33.935768 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:33.935971 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 12:17:33.936027 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 12:17:33.936237 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:33.928384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.3:48923)
0925 12:17:33.928559 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:33.928594 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:33.928984 (+ 390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:33.929191 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:33.929194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:33.929711 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:33.929867 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:33.929870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:33.930669 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:33.930670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:33.931770 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:33.931772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:33.932077 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:33.932078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:33.932163 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:33.932646 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:33.932657 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:33.933924 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:33.935072 (+ 1148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:33.935076 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:33.935078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:33.935257 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:33.935441 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:33.935441 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:33.935441 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:33.935524 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:33.935769 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:33.935771 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:33.935864 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 12:17:33.935941 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 12:17:33.936040 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":110,"threads_started":1}
I20260925 12:17:33.936661 6949 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40821
I20260925 12:17:33.936728 6949 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:33.936841 6949 heartbeater.cc:511] Master 127.0.98.62:40821 requested a full tablet report, sending...
I20260925 12:17:33.937140 6490 ts_manager.cc:194] Registered new tserver with Master: ca1f60f9f9c040bdaf2f95beb2142646 (127.0.98.3:37931)
I20260925 12:17:33.937476 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48923
I20260925 12:17:33.942114 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260925 12:17:33.963564 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43889
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40821,127.0.98.61:43889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:34.038354 6954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:34.038559 6954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:34.038594 6954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:34.039697 6954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:34.039745 6954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:34.039767 6954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:34.039824 6954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:34.039847 6954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:17:34.041090 6954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40821,127.0.98.61:43889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
W20260925 12:17:34.041227 6954 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:40821,127.0.98.61:43889'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 12:17:34.041369 6954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:34.041584 6954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:34.045951 6959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:34.045995 6954 server_base.cc:1034] running on GCE node
W20260925 12:17:34.045948 6962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:34.045949 6960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:34.046275 6954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:34.046522 6954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:34.047664 6954 hybrid_clock.cc:648] HybridClock initialized: now 1790338654047659 us; error 39 us; skew 500 ppm
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:34.049770 6954 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260925 12:17:34.050117 6954 webserver.cc:460] Webserver started at http://127.0.98.61:37771/ using document root <none> and password file <none>
I20260925 12:17:34.050302 6954 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:34.050366 6954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:34.050491 6954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:34.051359 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/data/instance:
uuid: "11fca8bf24634bee88f5e4324d199660"
format_stamp: "Formatted at 2026-09-25 12:17:34 on dist-test-slave-clmx"
server_key: "a78acd17746abdf74bfac851d5f34c01"
server_key_iv: "1286ca75fc98462728dfee2b43cc15c8"
server_key_version: "encryptionkey@0"
I20260925 12:17:34.051718 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/wal/instance:
uuid: "11fca8bf24634bee88f5e4324d199660"
format_stamp: "Formatted at 2026-09-25 12:17:34 on dist-test-slave-clmx"
server_key: "a78acd17746abdf74bfac851d5f34c01"
server_key_iv: "1286ca75fc98462728dfee2b43cc15c8"
server_key_version: "encryptionkey@0"
I20260925 12:17:34.052866 6954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:34.053489 6969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:34.053649 6954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:34.053736 6954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/wal
uuid: "11fca8bf24634bee88f5e4324d199660"
format_stamp: "Formatted at 2026-09-25 12:17:34 on dist-test-slave-clmx"
server_key: "a78acd17746abdf74bfac851d5f34c01"
server_key_iv: "1286ca75fc98462728dfee2b43cc15c8"
server_key_version: "encryptionkey@0"
I20260925 12:17:34.053812 6954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:34.068203 6954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:34.069628 6954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:34.069757 6954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:34.073617 6954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43889
I20260925 12:17:34.073691 7021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43889 every 8 connection(s)
I20260925 12:17:34.073954 6954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/data/info.pb
I20260925 12:17:34.074877 7022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:34.076228 7022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:17:34.077241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 6954
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 12:17:34.085124 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.076790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.0.98.62:40821)
0925 12:17:34.076856 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.076858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.076996 (+ 138us) server_negotiation.cc:407] Connection header received
0925 12:17:34.077443 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.077444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.077462 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.077523 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.078234 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.078705 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.079708 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.079893 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.079970 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.082078 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.082079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.082079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.082086 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.083710 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.084180 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.084181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.084181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.084231 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.084484 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.084485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.084485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.084608 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.084692 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.084941 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.085021 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 12:17:34.085279 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.076756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.0.1:47326)
0925 12:17:34.076899 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.076927 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.077289 (+ 362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.077598 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.077602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.077956 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.078154 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.078158 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.078814 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.078815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.079599 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.079601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.079977 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.079977 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.080081 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.080513 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.080523 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.082001 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.083788 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.083794 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.083796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.084110 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.084305 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.084306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.084307 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.084424 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.084683 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.084688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.084813 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.084909 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.085036 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":138,"thread_start_us":75,"threads_started":1}
I20260925 12:17:34.085750 7022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:17:34.086022 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.077715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40404 (local address 127.0.98.61:43889)
0925 12:17:34.077873 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.077877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.077894 (+ 17us) server_negotiation.cc:407] Connection header received
0925 12:17:34.077912 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.077914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.077936 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.078012 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.078511 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.079276 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.079967 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.080268 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.080332 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.082675 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.082681 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.082681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.082687 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.084582 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.084886 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.084889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.084889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.084939 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.085156 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.085157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.085158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.085648 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.085767 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.085845 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.085919 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 12:17:34.090986 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.085959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43889 (local address 127.0.0.1:40408)
0925 12:17:34.085994 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.086001 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.086031 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.086293 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.086293 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.086404 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.086443 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.086444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.087025 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.087026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.087446 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.087447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.087795 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.087796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.087842 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.088198 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.088203 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.089201 (+ 998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.090327 (+ 1126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.090328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.090328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.090481 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.090624 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.090624 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.090625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.090666 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.090807 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.090807 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.090833 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.090879 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.090937 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 12:17:34.091038 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.086010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40408 (local address 127.0.98.61:43889)
0925 12:17:34.086126 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.086128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.086135 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:17:34.086153 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.086153 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.086163 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.086208 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.086515 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.086965 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.087509 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.087740 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.087787 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.089266 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.089267 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.089267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.089272 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.090266 (+ 994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.090537 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.090537 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.090538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.090575 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.090700 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.090700 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.090700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.090765 (+ 65us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.090801 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.090894 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.090973 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260925 12:17:34.092052 7022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: Bootstrap starting.
I20260925 12:17:34.092779 7022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:34.093032 7022 log.cc:824] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:34.093641 7022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: No bootstrap required, opened a new log
I20260925 12:17:34.095101 7022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } }
I20260925 12:17:34.095223 7022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:34.095263 7022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11fca8bf24634bee88f5e4324d199660, State: Initialized, Role: FOLLOWER
I20260925 12:17:34.095381 7022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } }
I20260925 12:17:34.095671 7022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:34.095654 7029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } } }
I20260925 12:17:34.095793 7029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 12:17:34.097185 7040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:17:34.097335 7040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:17:34.098503 6954 master_runner.cc:428] Detected that this master 11fca8bf24634bee88f5e4324d199660 is joining an existing cluster
I20260925 12:17:34.098556 6954 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:43889
I20260925 12:17:34.105594 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.099413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43889 (local address 127.0.0.1:40420)
0925 12:17:34.099558 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.099567 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.099633 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.099889 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.099889 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.100073 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.100122 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.100123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.100625 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.100626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.101404 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.101406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.101868 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.101868 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.101904 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.102352 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.102356 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.102669 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.104594 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.104595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.104595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.104804 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.104982 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.104983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.104983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.105043 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.105281 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.105282 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.105320 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.105394 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.105490 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260925 12:17:34.105788 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.099573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.0.98.61:43889)
0925 12:17:34.099607 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.099609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.099645 (+ 36us) server_negotiation.cc:407] Connection header received
0925 12:17:34.099815 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.099816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.099825 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.099869 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.100169 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.100559 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.101488 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.101758 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.101791 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.102767 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.102768 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.102769 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.102775 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.104509 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.104869 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.104870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.104870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.104911 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.105095 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.105096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.105096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.105207 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.105269 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.105424 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.105545 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":19}
I20260925 12:17:34.105966 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.099383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.0.98.62:40821)
0925 12:17:34.099418 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.099419 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.099568 (+ 149us) server_negotiation.cc:407] Connection header received
0925 12:17:34.099695 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.099696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.099708 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.099779 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.100120 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.100767 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.101768 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.102061 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.102133 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.103212 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.103214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.103214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.103221 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.104723 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.105055 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.105056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.105056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.105104 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.105313 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.105314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.105315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.105434 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.105501 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.105789 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.105874 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:17:34.106371 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.099357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.0.1:47330)
0925 12:17:34.099485 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.099494 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.099591 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.099780 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.099780 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.099962 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.100045 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.100046 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.100841 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.100842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.101691 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.101692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.102140 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.102141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.102204 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.102813 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.102817 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.103148 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.104792 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.104793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.104794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.104996 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.105171 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.105171 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.105172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.105234 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.105612 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.105613 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.105660 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.106055 (+ 395us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.106154 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338653, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 12:17:34.114724 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.108321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.0.98.61:43889)
0925 12:17:34.108368 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.108369 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.108510 (+ 141us) server_negotiation.cc:407] Connection header received
0925 12:17:34.108557 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.108558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.108566 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.108633 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.108938 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.109593 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.110291 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.110468 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.110532 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.112437 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.112438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.112438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.112445 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.113634 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.113995 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.113996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.113996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.114045 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.114227 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.114228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.114228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.114317 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.114399 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.114580 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.114645 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 12:17:34.114748 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.108273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43889 (local address 127.0.0.1:40426)
0925 12:17:34.108410 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.108445 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.108507 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.108643 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.108646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.108838 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.108876 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.108878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.109660 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.109661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.110177 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.110220 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.110533 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.110534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.110611 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.110983 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.110987 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.112336 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.113711 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.113718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.113720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.113912 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.114107 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.114107 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.114108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.114168 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.114396 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.114403 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.114449 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.114542 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.114647 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260925 12:17:34.115149 6490 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:43889: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "11fca8bf24634bee88f5e4324d199660"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 43889
}
attrs {
promote: true
}
}
dest_uuid: "8b159ed78d924726b4d7a669d916a880"
cas_config_opid_index: -1
I20260925 12:17:34.115422 6490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: true } }
I20260925 12:17:34.115839 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 11fca8bf24634bee88f5e4324d199660 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: true } } }
I20260925 12:17:34.116098 6996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:34.116945 6996 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 1 FOLLOWER]: Refusing update from remote peer 8b159ed78d924726b4d7a669d916a880: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 12:17:34.117022 7050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:43889
I20260925 12:17:34.117220 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:34.117443 6954 master.cc:537] Master@127.0.98.61:43889 shutting down...
I20260925 12:17:34.117604 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8b159ed78d924726b4d7a669d916a880" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: true } } }
I20260925 12:17:34.117733 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: This master's current role is: LEADER
W20260925 12:17:34.117861 6476 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 12:17:34.118273 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 -> Peer 11fca8bf24634bee88f5e4324d199660 (127.0.98.61:43889): Couldn't send request to peer 11fca8bf24634bee88f5e4324d199660. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 12:17:34.119963 6954 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:34.120064 6954 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:34.120108 6954 tablet_replica.cc:330] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: stopping tablet replica
I20260925 12:17:34.132144 6954 master.cc:552] Master@127.0.98.61:43889 shutdown complete.
I20260925 12:17:34.132210 6954 master_runner.cc:305] Clearing existing system tablet
I20260925 12:17:34.132314 6954 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:34.132385 6954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:34.132997 6954 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:34.133453 7055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:34.133620 6954 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:34.133677 6954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:34.133701 6954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/wal
uuid: "11fca8bf24634bee88f5e4324d199660"
format_stamp: "Formatted at 2026-09-25 12:17:34 on dist-test-slave-clmx"
server_key: "a78acd17746abdf74bfac851d5f34c01"
server_key_iv: "1286ca75fc98462728dfee2b43cc15c8"
server_key_version: "encryptionkey@0"
I20260925 12:17:34.134361 6954 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:34.135967 6954 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 12:17:34.136029 6954 log.cc:1196] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 12:17:34.136216 6954 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: Deleting consensus metadata
I20260925 12:17:34.136318 6954 master_runner.cc:315] Copying system tablet from 127.0.98.62:40821
I20260925 12:17:34.136818 6954 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:40821
I20260925 12:17:34.142262 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.137107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47338 (local address 127.0.98.62:40821)
0925 12:17:34.137143 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.137145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.137239 (+ 94us) server_negotiation.cc:407] Connection header received
0925 12:17:34.137279 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.137279 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.137287 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.137348 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.137558 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.138220 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.138848 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.139149 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.139213 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.140115 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.140117 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.140117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.140123 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.141304 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.141609 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.141610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.141610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.141662 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.141833 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.141833 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.141834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.141936 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.142009 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.142121 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.142181 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:17:34.142295 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.137046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.0.1:47338)
0925 12:17:34.137135 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.137144 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.137231 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.137351 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.137352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.137461 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.137501 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.137502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.138307 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.138308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.138778 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.138779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.139216 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.139216 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.139257 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.139720 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.139724 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.139942 (+ 218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.141396 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.141397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.141397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.141545 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.141721 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.141721 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.141722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.141770 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.142001 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.142002 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.142038 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.142097 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.142182 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":59,"threads_started":1}
I20260925 12:17:34.142534 6510 tablet_copy_service.cc:140] P 8b159ed78d924726b4d7a669d916a880: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 11fca8bf24634bee88f5e4324d199660 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47338)
I20260925 12:17:34.142622 6510 tablet_copy_service.cc:161] P 8b159ed78d924726b4d7a669d916a880: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 11fca8bf24634bee88f5e4324d199660 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47338: session id = 11fca8bf24634bee88f5e4324d199660-00000000000000000000000000000000
I20260925 12:17:34.143133 6510 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 12:17:34.143734 6954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 12:17:34.144766 6954 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: tablet copy: Starting download of 0 data blocks...
I20260925 12:17:34.144909 6954 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: tablet copy: Starting download of 1 WAL segments...
I20260925 12:17:34.145709 6954 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 12:17:34.146493 6510 tablet_copy_service.cc:342] P 8b159ed78d924726b4d7a669d916a880: Request end of tablet copy session 11fca8bf24634bee88f5e4324d199660-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47338
I20260925 12:17:34.146550 6510 tablet_copy_service.cc:433] P 8b159ed78d924726b4d7a669d916a880: ending tablet copy session 11fca8bf24634bee88f5e4324d199660-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 11fca8bf24634bee88f5e4324d199660
I20260925 12:17:34.148783 6954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:34.149714 7063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:34.149726 7062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:34.149808 6954 server_base.cc:1034] running on GCE node
W20260925 12:17:34.156236 7065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:34.156389 6954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:34.156517 6954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:34.157644 6954 hybrid_clock.cc:648] HybridClock initialized: now 1790338654157633 us; error 25 us; skew 500 ppm
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:34.158712 6954 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260925 12:17:34.158993 6954 webserver.cc:460] Webserver started at http://127.0.98.61:45791/ using document root <none> and password file <none>
I20260925 12:17:34.159093 6954 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:34.159140 6954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:34.159677 6954 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:34.160194 7072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:34.160305 6954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:34.160347 6954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/wal
uuid: "11fca8bf24634bee88f5e4324d199660"
format_stamp: "Formatted at 2026-09-25 12:17:34 on dist-test-slave-clmx"
server_key: "a78acd17746abdf74bfac851d5f34c01"
server_key_iv: "1286ca75fc98462728dfee2b43cc15c8"
server_key_version: "encryptionkey@0"
I20260925 12:17:34.160395 6954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:34.169718 6954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:34.171205 6954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:34.174098 6954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43889
I20260925 12:17:34.174234 7124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43889 every 8 connection(s)
I20260925 12:17:34.174412 6954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-1/data/info.pb
I20260925 12:17:34.175137 7125 sys_catalog.cc:263] Verifying existing consensus state
I20260925 12:17:34.175581 7125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: Bootstrap starting.
I20260925 12:17:34.178117 7125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:17:34.178351 7125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: Bootstrap complete.
I20260925 12:17:34.178522 7125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: true } }
I20260925 12:17:34.178593 7125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 11fca8bf24634bee88f5e4324d199660, State: Initialized, Role: LEARNER
I20260925 12:17:34.178627 7125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: true } }
I20260925 12:17:34.178817 7125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:34.178822 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: true } } }
I20260925 12:17:34.178951 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [sys.catalog]: This master's current role is: LEARNER
I20260925 12:17:34.180218 7139 catalog_manager.cc:1259] Loaded cluster ID: 9ebbedcdd3ba4f3ca9c70ee49c9b30d5
I20260925 12:17:34.180298 7139 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: loading cluster ID for follower catalog manager: success
I20260925 12:17:34.181169 7139 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: acquiring CA information for follower catalog manager: success
I20260925 12:17:34.181406 7139 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 12:17:34.193406 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.187654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.0.98.61:43889)
0925 12:17:34.187749 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.187751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.187759 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:34.187777 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.187778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.187787 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.187857 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.188324 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.188928 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.189619 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.189814 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.189896 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.190961 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.190963 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.190963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.190969 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.192341 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.192689 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.192690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.192690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.192745 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.192937 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.192938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.192939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.193037 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.193136 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.193233 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.193305 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":43,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260925 12:17:34.214458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38175
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40821,127.0.98.61:43889,127.0.98.60:38175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:34.289527 7143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:34.289697 7143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:34.289717 7143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:34.290714 7143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:34.290747 7143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:34.290760 7143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:34.290772 7143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:34.290784 7143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:17:34.292111 7143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40821,127.0.98.61:43889,127.0.98.60:38175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:34.292287 7143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:34.292444 7143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:34.296962 7148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:34.296962 7151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:34.297036 7149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:34.297236 7143 server_base.cc:1034] running on GCE node
I20260925 12:17:34.297487 7143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:34.297757 7143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:34.298911 7143 hybrid_clock.cc:648] HybridClock initialized: now 1790338654298909 us; error 34 us; skew 500 ppm
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:34.301162 7143 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260925 12:17:34.301607 7143 webserver.cc:460] Webserver started at http://127.0.98.60:38611/ using document root <none> and password file <none>
I20260925 12:17:34.301812 7143 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:34.301879 7143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:34.302033 7143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:34.302991 7143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/data/instance:
uuid: "b8ea31e1934c4a6da6f93396cbe910dc"
format_stamp: "Formatted at 2026-09-25 12:17:34 on dist-test-slave-clmx"
server_key: "12005587d9c6e227efb62e5eb83bbf99"
server_key_iv: "462c6a678fd0e130b47e7cf2b6449b95"
server_key_version: "encryptionkey@0"
I20260925 12:17:34.303330 7143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/wal/instance:
uuid: "b8ea31e1934c4a6da6f93396cbe910dc"
format_stamp: "Formatted at 2026-09-25 12:17:34 on dist-test-slave-clmx"
server_key: "12005587d9c6e227efb62e5eb83bbf99"
server_key_iv: "462c6a678fd0e130b47e7cf2b6449b95"
server_key_version: "encryptionkey@0"
I20260925 12:17:34.304600 7143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:17:34.305305 7158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:34.305469 7143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:34.305558 7143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/wal
uuid: "b8ea31e1934c4a6da6f93396cbe910dc"
format_stamp: "Formatted at 2026-09-25 12:17:34 on dist-test-slave-clmx"
server_key: "12005587d9c6e227efb62e5eb83bbf99"
server_key_iv: "462c6a678fd0e130b47e7cf2b6449b95"
server_key_version: "encryptionkey@0"
I20260925 12:17:34.305637 7143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:34.315481 7143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:34.317126 7143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:34.317276 7143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:34.320976 7143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38175
I20260925 12:17:34.321283 7143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/data/info.pb
I20260925 12:17:34.321484 7210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38175 every 8 connection(s)
I20260925 12:17:34.322206 7211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:34.323565 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:17:34.328637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 7143
I20260925 12:17:34.331868 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.324374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47342 (local address 127.0.98.62:40821)
0925 12:17:34.324413 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.324415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.324506 (+ 91us) server_negotiation.cc:407] Connection header received
0925 12:17:34.324892 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.324893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.324902 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.324954 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.325460 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.325912 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.326780 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.326954 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.327001 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.329044 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.329046 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.329046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.329054 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.330532 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.330984 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.330985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.330985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.331033 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.331266 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.331267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.331267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.331354 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.331405 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.331720 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.331788 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:17:34.331988 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.324238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.0.1:47342)
0925 12:17:34.324379 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.324425 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.324818 (+ 393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.325001 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.325004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.325215 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.325394 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.325398 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.325982 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.325983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.326711 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.326713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.327017 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.327018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.327101 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.327553 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.327560 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.328925 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.330628 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.330634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.330635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.330912 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.331092 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.331093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.331093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.331209 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.331409 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.331413 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.331550 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.331695 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.331806 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260925 12:17:34.332568 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 12:17:34.337330 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.329087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54374 (local address 127.0.98.60:38175)
0925 12:17:34.329225 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.329228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.329246 (+ 18us) server_negotiation.cc:407] Connection header received
0925 12:17:34.329261 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.329263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.329271 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.329340 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.329926 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.330667 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.331531 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.331910 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.332021 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.334699 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.334704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.334705 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.334711 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.336186 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.336570 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.336572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.336573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.336630 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.336824 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.336825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.336825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.336994 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.337145 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.337223 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.337274 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":94,"threads_started":1}
I20260925 12:17:34.338969 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.332810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43889 (local address 127.0.0.1:40456)
0925 12:17:34.332939 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.332947 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.332977 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.333110 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.333110 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.333209 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.333244 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.333245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.334050 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.334050 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.334633 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.334635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.335048 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.335049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.335125 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.335479 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.335483 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.336655 (+ 1172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.338268 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.338269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.338269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.338406 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.338537 (+ 131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.338537 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.338537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.338573 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.338785 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.338786 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.338813 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.338868 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.338922 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 12:17:34.339026 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.332905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40456 (local address 127.0.98.61:43889)
0925 12:17:34.332955 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.332958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.332969 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:17:34.333024 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.333025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.333039 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.333111 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.333309 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.333957 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.334738 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.334947 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.335047 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.336732 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.336734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.336734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.336740 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.338210 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.338460 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.338460 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.338460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.338501 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.338605 (+ 104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.338606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.338606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.338729 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.338789 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.338891 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.338949 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 12:17:34.339336 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 12:17:34.344554 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.339566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.0.1:54386)
0925 12:17:34.339645 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.339655 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.339703 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.339795 (+ 92us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.339796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.339891 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.339912 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.339913 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.340442 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.340443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.340883 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.340884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.341171 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.341171 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.341201 (+ 30us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.341589 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.341598 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.342558 (+ 960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.343835 (+ 1277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.343836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.343836 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.343982 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.344162 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.344163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.344163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.344208 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.344372 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.344373 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.344400 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.344441 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.344504 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 12:17:34.344612 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.339621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54386 (local address 127.0.98.60:38175)
0925 12:17:34.339669 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.339682 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.339692 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:17:34.339751 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.339752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.339758 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.339793 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.339960 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.340374 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.340944 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.341110 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.341155 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.342628 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.342629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.342629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.342634 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.343759 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.344049 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.344050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.344050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.344092 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.344254 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.344254 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.344255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.344329 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.344370 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.344464 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.344545 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":34}
I20260925 12:17:34.345563 7211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc: Bootstrap starting.
I20260925 12:17:34.346287 7211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:34.346534 7211 log.cc:824] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:34.347165 7211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc: No bootstrap required, opened a new log
I20260925 12:17:34.348462 7211 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } }
I20260925 12:17:34.348598 7211 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:34.348640 7211 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ea31e1934c4a6da6f93396cbe910dc, State: Initialized, Role: FOLLOWER
I20260925 12:17:34.348752 7211 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } }
I20260925 12:17:34.349009 7211 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:34.348989 7216 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } } }
I20260925 12:17:34.349130 7216 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [sys.catalog]: This master's current role is: FOLLOWER
W20260925 12:17:34.350479 7227 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:17:34.350528 7227 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:17:34.352476 7143 master_runner.cc:428] Detected that this master b8ea31e1934c4a6da6f93396cbe910dc is joining an existing cluster
I20260925 12:17:34.352530 7143 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:38175
I20260925 12:17:34.360117 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.353378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.0.1:47346)
0925 12:17:34.354170 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.354180 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.354257 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.354432 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.354432 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.354656 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.354715 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.354717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.355608 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.355608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.356501 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.356503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.356977 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.356977 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.357027 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.357409 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.357414 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.357696 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.359079 (+ 1383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.359080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.359080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.359238 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.359442 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.359443 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.359443 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.359496 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.359770 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.359770 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.359815 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.359929 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.360022 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":1,"thread_start_us":728,"threads_started":1}
I20260925 12:17:34.360137 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.353414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47346 (local address 127.0.98.62:40821)
0925 12:17:34.353453 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.353455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.354235 (+ 780us) server_negotiation.cc:407] Connection header received
0925 12:17:34.354332 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.354333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.354344 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.354392 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.354800 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.355522 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.356590 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.356866 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.356915 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.357798 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.357800 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.357800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.357806 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.358987 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.359317 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.359318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.359318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.359367 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.359564 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.359565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.359565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.359719 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.359776 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.359956 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.360035 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338653, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 12:17:34.382784 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.361965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.0.1:54406)
0925 12:17:34.362017 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.362026 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.362058 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.362239 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.362239 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.362379 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.362434 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.362435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.363052 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.363054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.363541 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.363543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.363960 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.363961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.364032 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.364402 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.364406 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.365615 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.381697 (+ 16082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.381698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.381698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.381915 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.382124 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.382125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.382125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.382180 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.382476 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.382477 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.382521 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.382570 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.382665 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 12:17:34.383116 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.362001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54406 (local address 127.0.98.60:38175)
0925 12:17:34.362117 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.362119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.362125 (+ 6us) server_negotiation.cc:407] Connection header received
0925 12:17:34.362152 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.362153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.362161 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.362236 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.362515 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.362973 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.363659 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.363873 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.363957 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.365686 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.365687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.365687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.365692 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.381602 (+ 15910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.381999 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.382000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.382001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.382053 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.382250 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.382251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.382251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.382365 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.382881 (+ 516us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.382921 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.383003 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
I20260925 12:17:34.383536 6490 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:38175: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 38175
}
attrs {
promote: true
}
}
dest_uuid: "8b159ed78d924726b4d7a669d916a880"
cas_config_opid_index: 5
I20260925 12:17:34.383798 6490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: true } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: true } }
I20260925 12:17:34.383857 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.353971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40462 (local address 127.0.98.61:43889)
0925 12:17:34.354024 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.354026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.354300 (+ 274us) server_negotiation.cc:407] Connection header received
0925 12:17:34.354410 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.354411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.354419 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.354447 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.354898 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.355583 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.356648 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.356920 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.356960 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.380641 (+ 23681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.380642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.380643 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.380649 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.382605 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.382976 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.382977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.382978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.383026 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.383308 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.383309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.383310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.383437 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.383477 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.383699 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.383771 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:17:34.384184 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.353866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43889 (local address 127.0.0.1:40462)
0925 12:17:34.354240 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.354249 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.354331 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.354501 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.354501 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.354753 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.354809 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.354810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.355697 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.355698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.356551 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.356553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.357028 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.357028 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.357085 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.380042 (+ 22957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.380046 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.380552 (+ 506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.382697 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.382698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.382699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.382895 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.383102 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.383103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.383103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.383155 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.383523 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.383523 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.383573 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.383952 (+ 379us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.384073 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":33,"thread_start_us":220,"threads_started":1}
I20260925 12:17:34.384235 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b8ea31e1934c4a6da6f93396cbe910dc (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: true } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: true } } }
I20260925 12:17:34.384359 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.354063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.0.98.60:38175)
0925 12:17:34.354097 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.354099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.354422 (+ 323us) server_negotiation.cc:407] Connection header received
0925 12:17:34.354495 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.354496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.354503 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.354526 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.354918 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.355566 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.356586 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.356839 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.356872 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.380870 (+ 23998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.380871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.380871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.380877 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.382474 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.382793 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.382794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.382794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.382837 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.383500 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.383500 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.383501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.383617 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.383680 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.384219 (+ 539us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.384285 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:17:34.385042 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.354083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.0.1:54390)
0925 12:17:34.354366 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.354375 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.354438 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.354574 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.354574 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.354805 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.354850 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.354851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.355657 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.355658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.356501 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.356503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.356945 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.356946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.356996 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.380342 (+ 23346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.380346 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.380783 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.382546 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.382546 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.382547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.382724 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.383360 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.383361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.383361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.383430 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.383812 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.383813 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.383849 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.384801 (+ 952us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.384906 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":32,"thread_start_us":112,"threads_started":1}
I20260925 12:17:34.385098 7050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:38175
I20260925 12:17:34.385156 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8b159ed78d924726b4d7a669d916a880" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: true } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: true } } }
I20260925 12:17:34.385227 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:34.385824 7143 master.cc:537] Master@127.0.98.60:38175 shutting down...
I20260925 12:17:34.385959 7185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:34.386780 7185 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [term 1 FOLLOWER]: Refusing update from remote peer 8b159ed78d924726b4d7a669d916a880: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 12:17:34.388151 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:34.388963 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.385460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43889 (local address 127.0.0.1:40468)
0925 12:17:34.385778 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.385788 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.385826 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.386016 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.386018 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.386019 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:34.386050 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.386051 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.386876 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.386877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.387920 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.387922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.388702 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.388703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.388762 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.388782 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.388833 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.388876 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305}
I20260925 12:17:34.389304 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.385527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40468 (local address 127.0.98.61:43889)
0925 12:17:34.385559 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.385561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.385877 (+ 316us) server_negotiation.cc:407] Connection header received
0925 12:17:34.385897 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.385898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.385916 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.385966 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:34.386122 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.386786 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.388036 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.388615 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.388691 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.388753 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.389187 (+ 434us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.389228 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 12:17:34.389488 7099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 1 LEARNER]: Refusing update from remote peer 8b159ed78d924726b4d7a669d916a880: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260925 12:17:34.389637 6475 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 -> Peer b8ea31e1934c4a6da6f93396cbe910dc (127.0.98.60:38175): Couldn't send request to peer b8ea31e1934c4a6da6f93396cbe910dc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 12:17:34.389763 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 12:17:34.390144 7143 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:34.390220 7143 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:34.390265 7143 tablet_replica.cc:330] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc: stopping tablet replica
I20260925 12:17:34.390239 7099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b8ea31e1934c4a6da6f93396cbe910dc (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: true } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: true } } }
I20260925 12:17:34.391207 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b159ed78d924726b4d7a669d916a880" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: true } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: true } } }
I20260925 12:17:34.391316 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [sys.catalog]: This master's current role is: LEARNER
I20260925 12:17:34.391201 7238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b159ed78d924726b4d7a669d916a880. Latest consensus state: current_term: 1 leader_uuid: "8b159ed78d924726b4d7a669d916a880" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: true } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: true } } }
I20260925 12:17:34.391441 7238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [sys.catalog]: This master's current role is: LEARNER
I20260925 12:17:34.391700 7051 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880: attempting to promote NON_VOTER 11fca8bf24634bee88f5e4324d199660 to VOTER
I20260925 12:17:34.391887 7051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: true } }
I20260925 12:17:34.392230 7099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 1 LEARNER]: Refusing update from remote peer 8b159ed78d924726b4d7a669d916a880: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 12:17:34.392278 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.392188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.0.1:54410)
0925 12:17:34.392217 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.392256 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 12:17:34.392424 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 12:17:34.392845 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.392550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.0.1:54414)
0925 12:17:34.392746 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.392775 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260925 12:17:34.392961 7049 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 11fca8bf24634bee88f5e4324d199660 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: true } } }
W20260925 12:17:34.393043 6475 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 -> Peer b8ea31e1934c4a6da6f93396cbe910dc (127.0.98.60:38175): Couldn't send request to peer b8ea31e1934c4a6da6f93396cbe910dc. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38175: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 12:17:34.393771 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8b159ed78d924726b4d7a669d916a880" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: true } } }
I20260925 12:17:34.393802 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b159ed78d924726b4d7a669d916a880" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: true } } }
I20260925 12:17:34.393859 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:34.393877 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:34.393977 7099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 11fca8bf24634bee88f5e4324d199660 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: true } } }
I20260925 12:17:34.394791 7238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b159ed78d924726b4d7a669d916a880" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: true } } }
I20260925 12:17:34.394879 7238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:17:34.402987 7143 master.cc:552] Master@127.0.98.60:38175 shutdown complete.
I20260925 12:17:34.403033 7143 master_runner.cc:305] Clearing existing system tablet
I20260925 12:17:34.403154 7143 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:34.403198 7143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:34.403888 7143 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.001s
I20260925 12:17:34.404459 7240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:34.404610 7143 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:34.404676 7143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:34.404703 7143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/wal
uuid: "b8ea31e1934c4a6da6f93396cbe910dc"
format_stamp: "Formatted at 2026-09-25 12:17:34 on dist-test-slave-clmx"
server_key: "12005587d9c6e227efb62e5eb83bbf99"
server_key_iv: "462c6a678fd0e130b47e7cf2b6449b95"
server_key_version: "encryptionkey@0"
I20260925 12:17:34.405373 7143 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:34.406792 7143 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 12:17:34.406858 7143 log.cc:1196] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 12:17:34.407043 7143 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc: Deleting consensus metadata
I20260925 12:17:34.407160 7143 master_runner.cc:315] Copying system tablet from 127.0.98.62:40821
I20260925 12:17:34.407763 7143 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:40821
I20260925 12:17:34.413902 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.408043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47348 (local address 127.0.98.62:40821)
0925 12:17:34.408079 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.408080 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.408127 (+ 47us) server_negotiation.cc:407] Connection header received
0925 12:17:34.408187 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.408188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.408194 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.408230 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.408562 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.409050 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.410031 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.410212 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.410262 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.411499 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.411500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.411500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.411505 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.412931 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.413290 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.413291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.413291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.413335 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.413513 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.413514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.413514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.413613 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.413658 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.413800 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.413846 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:17:34.413947 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.407964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.0.1:47348)
0925 12:17:34.408070 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.408079 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.408136 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.408243 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.408244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.408399 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.408503 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.408504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.409114 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.409115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.409966 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.409968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.410279 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.410280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.410341 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.411092 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.411097 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.411424 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.413009 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.413010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.413010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.413216 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.413392 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.413393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.413393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.413457 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.413666 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.413667 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.413705 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.413785 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.413864 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":71,"threads_started":1}
I20260925 12:17:34.414448 6510 tablet_copy_service.cc:140] P 8b159ed78d924726b4d7a669d916a880: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b8ea31e1934c4a6da6f93396cbe910dc ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47348)
I20260925 12:17:34.414510 6510 tablet_copy_service.cc:161] P 8b159ed78d924726b4d7a669d916a880: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b8ea31e1934c4a6da6f93396cbe910dc at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47348: session id = b8ea31e1934c4a6da6f93396cbe910dc-00000000000000000000000000000000
I20260925 12:17:34.414891 6510 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 12:17:34.415308 7143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 12:17:34.416306 7143 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc: tablet copy: Starting download of 0 data blocks...
I20260925 12:17:34.416469 7143 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc: tablet copy: Starting download of 1 WAL segments...
I20260925 12:17:34.417156 7143 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 12:17:34.417896 6510 tablet_copy_service.cc:342] P 8b159ed78d924726b4d7a669d916a880: Request end of tablet copy session b8ea31e1934c4a6da6f93396cbe910dc-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47348
I20260925 12:17:34.417949 6510 tablet_copy_service.cc:433] P 8b159ed78d924726b4d7a669d916a880: ending tablet copy session b8ea31e1934c4a6da6f93396cbe910dc-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b8ea31e1934c4a6da6f93396cbe910dc
I20260925 12:17:34.420046 7143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:34.421162 7250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:34.421168 7247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:34.421185 7248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:34.423789 7143 server_base.cc:1034] running on GCE node
I20260925 12:17:34.423882 7143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:34.424014 7143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:34.425151 7143 hybrid_clock.cc:648] HybridClock initialized: now 1790338654425150 us; error 28 us; skew 500 ppm
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:34.426273 7143 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260925 12:17:34.426542 7143 webserver.cc:460] Webserver started at http://127.0.98.60:43821/ using document root <none> and password file <none>
I20260925 12:17:34.426643 7143 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:34.426693 7143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:34.427207 7143 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:17:34.427688 7257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:34.427802 7143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:34.427847 7143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/wal
uuid: "b8ea31e1934c4a6da6f93396cbe910dc"
format_stamp: "Formatted at 2026-09-25 12:17:34 on dist-test-slave-clmx"
server_key: "12005587d9c6e227efb62e5eb83bbf99"
server_key_iv: "462c6a678fd0e130b47e7cf2b6449b95"
server_key_version: "encryptionkey@0"
I20260925 12:17:34.427917 7143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:34.438235 7143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:34.439781 7143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:34.442478 7143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38175
I20260925 12:17:34.442545 7309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38175 every 8 connection(s)
I20260925 12:17:34.442793 7143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/master-2/data/info.pb
I20260925 12:17:34.443392 7310 sys_catalog.cc:263] Verifying existing consensus state
I20260925 12:17:34.443852 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc: Bootstrap starting.
I20260925 12:17:34.445922 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:17:34.446156 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc: Bootstrap complete.
I20260925 12:17:34.446345 7310 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: true } }
I20260925 12:17:34.446409 7310 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b8ea31e1934c4a6da6f93396cbe910dc, State: Initialized, Role: LEARNER
I20260925 12:17:34.446434 7310 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: true } }
I20260925 12:17:34.446606 7310 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:34.446633 7313 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: true } } }
I20260925 12:17:34.446719 7313 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [sys.catalog]: This master's current role is: LEARNER
I20260925 12:17:34.447952 7324 catalog_manager.cc:1259] Loaded cluster ID: 9ebbedcdd3ba4f3ca9c70ee49c9b30d5
I20260925 12:17:34.448107 7324 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc: loading cluster ID for follower catalog manager: success
I20260925 12:17:34.448779 7324 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc: acquiring CA information for follower catalog manager: success
I20260925 12:17:34.449044 7324 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 12:17:34.498049 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.492192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.0.98.60:38175)
0925 12:17:34.492303 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.492305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.492313 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:34.492342 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.492342 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.492351 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.492446 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.492975 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.493766 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.494447 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.494687 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.494760 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.495911 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.495912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.495912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.495918 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.497026 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.497354 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.497356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.497356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.497408 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.497590 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.497591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.497592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.497692 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.497780 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.497880 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.497955 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":49,"threads_started":1}
I20260925 12:17:34.498517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 6552
I20260925 12:17:34.504257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38055
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40821
--webserver_port=42967
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40821,127.0.98.61:43889
--tserver_master_addrs=127.0.98.62:40821,127.0.98.61:43889,127.0.98.60:38175 with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:34.580160 7326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:34.580317 7326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:34.580336 7326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:34.581307 7326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:34.581341 7326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:34.581364 7326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:17:34.582672 7326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42967
--tserver_master_addrs=127.0.98.62:40821,127.0.98.61:43889,127.0.98.60:38175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:34.582827 7326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:34.582983 7326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:34.587507 7334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:34.587529 7332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:34.587795 7331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:34.587947 7326 server_base.cc:1034] running on GCE node
I20260925 12:17:34.588169 7326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:34.588369 7326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:34.589525 7326 hybrid_clock.cc:648] HybridClock initialized: now 1790338654589511 us; error 40 us; skew 500 ppm
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:34.591778 7326 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 12:17:34.592239 7326 webserver.cc:460] Webserver started at http://127.0.98.1:42967/ using document root <none> and password file <none>
I20260925 12:17:34.592476 7326 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:34.592554 7326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:34.593797 7326 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:34.594367 7341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:34.594534 7326 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:34.594622 7326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "80fa57d079294d4589eb3e970a421c77"
format_stamp: "Formatted at 2026-09-25 12:17:33 on dist-test-slave-clmx"
server_key: "ae3dcc6ed50cd101c552cfb25a698dbe"
server_key_iv: "028e080823e96ef9adfda5712b20d367"
server_key_version: "encryptionkey@0"
I20260925 12:17:34.594935 7326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:34.605769 7326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:34.607267 7326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:34.607395 7326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:34.607597 7326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:34.607870 7326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:34.607923 7326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:34.607959 7326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:34.607983 7326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:34.613315 7326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38055
I20260925 12:17:34.613348 7454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38055 every 8 connection(s)
I20260925 12:17:34.613665 7326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:17:34.618439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 7326
I20260925 12:17:34.618597 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 6686
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 12:17:34.623479 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39153
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40821
--webserver_port=44317
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40821,127.0.98.61:43889
--tserver_master_addrs=127.0.98.62:40821,127.0.98.61:43889,127.0.98.60:38175 with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:17:34.627358 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.614649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60873 (local address 127.0.98.62:40821)
0925 12:17:34.614721 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.614724 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.614771 (+ 47us) server_negotiation.cc:407] Connection header received
0925 12:17:34.615179 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.615181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.615193 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.615268 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.616015 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.616466 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.617765 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.618023 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.618074 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.623815 (+ 5741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.623816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.623817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.623825 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.626034 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.626392 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.626393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.626393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.626450 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.626668 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.626668 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.626669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.626826 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.626904 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.627202 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.627275 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 12:17:34.627547 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.614480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.1:60873)
0925 12:17:34.614719 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.614728 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.615081 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.615318 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.615322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.615764 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.615928 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.615932 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.616559 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.616560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.617686 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.617688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.618140 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.618141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.618216 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.620452 (+ 2236us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.620459 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.623743 (+ 3284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.626113 (+ 2370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.626114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.626115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.626326 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.626543 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.626544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.626544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.626601 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.626898 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.626903 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.627026 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.627129 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.627313 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":7,"thread_start_us":52,"threads_started":1}
I20260925 12:17:34.628026 7456 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40821
I20260925 12:17:34.628104 7456 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:34.628234 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.614254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45913 (local address 127.0.98.60:38175)
0925 12:17:34.614292 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.614294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.614499 (+ 205us) server_negotiation.cc:407] Connection header received
0925 12:17:34.615262 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.615264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.615277 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.615326 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.616031 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.616700 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.620268 (+ 3568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.620604 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.620673 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.623292 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.623293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.623293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.623301 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.624676 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.625278 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.625279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.625279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.625337 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.625622 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.625624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.625624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.625766 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.625839 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.628049 (+ 2210us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.628140 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 12:17:34.628270 7456 heartbeater.cc:511] Master 127.0.98.62:40821 requested a full tablet report, sending...
I20260925 12:17:34.628364 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.614803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38551 (local address 127.0.98.61:43889)
0925 12:17:34.614844 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.614846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.614891 (+ 45us) server_negotiation.cc:407] Connection header received
0925 12:17:34.615391 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.615392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.615403 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.615458 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.616030 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.616707 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.617764 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.618012 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.618074 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.625317 (+ 7243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.625319 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.625319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.625326 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.627095 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.627525 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.627526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.627527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.627584 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.627866 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.627867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.627867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.628024 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.628087 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.628214 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.628286 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:17:34.628497 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.614702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43889 (local address 127.0.98.1:38551)
0925 12:17:34.614844 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.614851 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.615098 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.615481 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.615481 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.615763 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.615929 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.615935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.616770 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.616771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.617686 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.617688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.618088 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.618089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.618217 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.621554 (+ 3337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.621559 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.625216 (+ 3657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.627188 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.627189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.627190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.627460 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.627692 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.627693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.627694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.627757 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.628079 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.628079 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.628119 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.628191 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.628278 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":5,"thread_start_us":62,"threads_started":1}
I20260925 12:17:34.628607 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.614260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.98.1:45913)
0925 12:17:34.614398 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.614431 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.615081 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.615345 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.615346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.615762 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.615929 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.615935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.617773 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.617773 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.618343 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.618345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.620970 (+ 2625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.620971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.621031 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.621409 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.621415 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.623200 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.624769 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.624776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.624778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.625177 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.625407 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.625408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.625408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.625520 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.627860 (+ 2340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.627860 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.627906 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.628308 (+ 402us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.628395 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260925 12:17:34.629021 6490 ts_manager.cc:194] Re-registered known tserver with Master: 80fa57d079294d4589eb3e970a421c77 (127.0.98.1:38055)
I20260925 12:17:34.629096 7455 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38175
I20260925 12:17:34.629128 7455 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:34.629180 7455 heartbeater.cc:511] Master 127.0.98.60:38175 requested a full tablet report, sending...
I20260925 12:17:34.629472 7274 ts_manager.cc:194] Registered new tserver with Master: 80fa57d079294d4589eb3e970a421c77 (127.0.98.1:38055)
I20260925 12:17:34.629479 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60873
I20260925 12:17:34.629580 7458 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43889
I20260925 12:17:34.629624 7458 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:34.629688 7458 heartbeater.cc:511] Master 127.0.98.61:43889 requested a full tablet report, sending...
I20260925 12:17:34.630048 7089 ts_manager.cc:194] Registered new tserver with Master: 80fa57d079294d4589eb3e970a421c77 (127.0.98.1:38055)
W20260925 12:17:34.700452 7462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:34.700603 7462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:34.700623 7462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:34.701570 7462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:34.701604 7462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:34.701648 7462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:17:34.702935 7462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44317
--tserver_master_addrs=127.0.98.62:40821,127.0.98.61:43889,127.0.98.60:38175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:34.703156 7462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:34.703370 7462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:34.707854 7470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:34.707850 7468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:34.707850 7467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:34.708032 7462 server_base.cc:1034] running on GCE node
I20260925 12:17:34.708206 7462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:34.708410 7462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:34.709569 7462 hybrid_clock.cc:648] HybridClock initialized: now 1790338654709534 us; error 36 us; skew 500 ppm
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:34.711807 7462 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 12:17:34.712163 7462 webserver.cc:460] Webserver started at http://127.0.98.2:44317/ using document root <none> and password file <none>
I20260925 12:17:34.712350 7462 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:34.712425 7462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:34.713639 7462 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:17:34.714167 7477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:34.714318 7462 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:17:34.714402 7462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "a05341f431ea4d129ca4e5d7afdbf1d2"
format_stamp: "Formatted at 2026-09-25 12:17:33 on dist-test-slave-clmx"
server_key: "713626eb73db78ed79c606b93e16de4c"
server_key_iv: "31f67222a9157c5b637db4b05c50a2ad"
server_key_version: "encryptionkey@0"
I20260925 12:17:34.714689 7462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:34.722971 7462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:34.725524 7462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:34.725670 7462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:34.725867 7462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:34.726158 7462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:34.726212 7462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:34.726246 7462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:34.726270 7462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:34.732024 7462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39153
I20260925 12:17:34.732092 7590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39153 every 8 connection(s)
I20260925 12:17:34.732375 7462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 12:17:34.738462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 7462
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 12:17:34.738608 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 6820
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:17:34.742048 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.732970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44195 (local address 127.0.98.61:43889)
0925 12:17:34.733012 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.733014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.733224 (+ 210us) server_negotiation.cc:407] Connection header received
0925 12:17:34.733939 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.733941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.733953 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.733986 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.734714 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.735386 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.736480 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.736688 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.736719 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.738601 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.738602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.738603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.738609 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.740477 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.740900 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.740901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.740901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.740971 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.741323 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.741325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.741325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.741453 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.741493 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.741828 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.741915 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:17:34.742777 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.732978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43889 (local address 127.0.98.2:44195)
0925 12:17:34.733117 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.733152 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.733804 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.734121 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.734124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.734483 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.734607 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.734612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.735472 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.735472 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.736383 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.736385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.736774 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.736775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.736815 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.737131 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.737134 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.738489 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.740546 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.740552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.740553 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.740832 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.741096 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.741097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.741097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.741242 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.741541 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.741547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.741690 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.742300 (+ 610us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.742510 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":68,"thread_start_us":84,"threads_started":1}
I20260925 12:17:34.743031 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.733421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44995 (local address 127.0.98.60:38175)
0925 12:17:34.733482 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.733484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.733493 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:17:34.733957 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.733958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.733968 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.733996 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.734700 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.735143 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.736113 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.736318 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.736406 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.739406 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.739407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.739407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.739413 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.741820 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.742175 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.742175 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.742176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.742227 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.742465 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.742466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.742466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.742594 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.742634 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.742848 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.742929 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 12:17:34.743297 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.733591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38643 (local address 127.0.98.62:40821)
0925 12:17:34.733666 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.733668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.733676 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:34.733944 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.733945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.733956 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.733986 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.734713 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.735383 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.736504 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.737219 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.737287 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.740532 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.740533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.740534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.740540 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.742109 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.742512 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.742513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.742513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.742560 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.742761 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.742762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.742762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.742897 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.742938 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.743146 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.743213 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 12:17:34.743292 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.732979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.98.2:44995)
0925 12:17:34.733129 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.733153 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.733815 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.734120 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.734123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.734407 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.734606 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.734609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.735236 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.735236 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.736020 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.736022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.736413 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.736414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.736574 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.737017 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.737026 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.739308 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.741905 (+ 2597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.741906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.741907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.742099 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.742293 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.742293 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.742294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.742339 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.742679 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.742679 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.742718 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.743106 (+ 388us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.743197 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":16,"thread_start_us":72,"threads_started":1}
I20260925 12:17:34.743640 7593 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43889
I20260925 12:17:34.743728 7593 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:34.743809 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.733286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.2:38643)
0925 12:17:34.733401 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.733409 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.733804 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.734121 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.734125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.734481 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.734607 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.734612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.735472 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.735472 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.736383 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.736385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.737300 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.737301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.737357 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.739081 (+ 1724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.739085 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.740436 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.742216 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.742217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.742218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.742416 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.742632 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.742633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.742633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.742687 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.742984 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.742985 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.743033 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.743411 (+ 378us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.743682 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":21,"thread_start_us":71,"threads_started":1}
I20260925 12:17:34.743897 7593 heartbeater.cc:511] Master 127.0.98.61:43889 requested a full tablet report, sending...
I20260925 12:17:34.744117 7592 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40821
I20260925 12:17:34.744165 7592 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:34.744230 7592 heartbeater.cc:511] Master 127.0.98.62:40821 requested a full tablet report, sending...
I20260925 12:17:34.744338 7089 ts_manager.cc:194] Registered new tserver with Master: a05341f431ea4d129ca4e5d7afdbf1d2 (127.0.98.2:39153)
I20260925 12:17:34.744429 6490 ts_manager.cc:194] Re-registered known tserver with Master: a05341f431ea4d129ca4e5d7afdbf1d2 (127.0.98.2:39153)
I20260925 12:17:34.744823 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38643
I20260925 12:17:34.745244 7591 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38175
I20260925 12:17:34.745280 7591 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:34.745461 7591 heartbeater.cc:511] Master 127.0.98.60:38175 requested a full tablet report, sending...
I20260925 12:17:34.745725 7274 ts_manager.cc:194] Registered new tserver with Master: a05341f431ea4d129ca4e5d7afdbf1d2 (127.0.98.2:39153)
I20260925 12:17:34.746894 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37931
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40821
--webserver_port=42427
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40821,127.0.98.61:43889
--tserver_master_addrs=127.0.98.62:40821,127.0.98.61:43889,127.0.98.60:38175 with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:17:34.818291 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.815528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.0.98.60:38175)
0925 12:17:34.815605 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.815607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.815617 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:17:34.815669 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.815670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.815691 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.815766 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:34.815878 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.816553 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.817452 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.818010 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.818102 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.818164 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.818190 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.818223 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 12:17:34.818302 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.815420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.0.1:54434)
0925 12:17:34.815494 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.815505 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.815551 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.815775 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.815776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.815778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:34.815817 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.815818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.816657 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.816657 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.817375 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.817378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.818104 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.818105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.818150 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.818154 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.818186 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.818216 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 12:17:34.818586 7284 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 12:17:34.818718 7313 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b159ed78d924726b4d7a669d916a880. Latest consensus state: current_term: 1 leader_uuid: "8b159ed78d924726b4d7a669d916a880" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: true } } }
I20260925 12:17:34.818809 7313 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [sys.catalog]: This master's current role is: LEARNER
W20260925 12:17:34.823361 7598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:34.823556 7598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:34.823593 7598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:34.824736 7598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:34.824793 7598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:34.824836 7598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:17:34.826200 7598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42427
--tserver_master_addrs=127.0.98.62:40821,127.0.98.61:43889,127.0.98.60:38175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:34.826438 7598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:34.826627 7598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:34.831120 7603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:34.831132 7604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:34.831359 7598 server_base.cc:1034] running on GCE node
W20260925 12:17:34.831130 7606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:34.831643 7598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:34.831867 7598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:34.833014 7598 hybrid_clock.cc:648] HybridClock initialized: now 1790338654833009 us; error 39 us; skew 500 ppm
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:34.835206 7598 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 12:17:34.835686 7598 webserver.cc:460] Webserver started at http://127.0.98.3:42427/ using document root <none> and password file <none>
I20260925 12:17:34.835870 7598 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:34.835940 7598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:34.837031 7598 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:17:34.837500 7613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:34.837634 7598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:34.837714 7598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "ca1f60f9f9c040bdaf2f95beb2142646"
format_stamp: "Formatted at 2026-09-25 12:17:33 on dist-test-slave-clmx"
server_key: "7720f8f7b91dd88cf1b11d95ee2bfe5f"
server_key_iv: "6b45d9222f7f42e2e3a5c094237746a7"
server_key_version: "encryptionkey@0"
I20260925 12:17:34.837970 7598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:34.846310 7598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:34.848165 7598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:34.848315 7598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:34.848527 7598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:34.848778 7598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:34.848806 7598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:34.848850 7598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:34.848875 7598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:34.854193 7598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37931
I20260925 12:17:34.854264 7726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37931 every 8 connection(s)
I20260925 12:17:34.854559 7598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 12:17:34.861202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 7598
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 12:17:34 dist-test-slave-clmx krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 12:17:34.865674 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.855169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.98.3:54745)
0925 12:17:34.855716 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.855725 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.856527 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.856749 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.856750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.857087 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.857180 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.857181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.858068 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.858069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.858938 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.858940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.859363 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.859364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.859424 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.859980 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.859985 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.862282 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.864024 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.864030 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.864032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.864261 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.864482 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.864483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.864484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.864644 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.865028 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.865034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.865194 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.865296 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.865463 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":29,"thread_start_us":85,"threads_started":1}
I20260925 12:17:34.865893 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.855239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48905 (local address 127.0.98.62:40821)
0925 12:17:34.855420 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.855422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.855859 (+ 437us) server_negotiation.cc:407] Connection header received
0925 12:17:34.856623 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.856624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.856634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.856666 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.857268 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.857972 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.859014 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.859279 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.859363 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.861868 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.861870 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.861870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.861878 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.864187 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.864558 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.864559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.864559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.864641 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.864850 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.864851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.864851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.864982 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.865039 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.865726 (+ 687us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.865799 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260925 12:17:34.865947 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.855420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54745 (local address 127.0.98.60:38175)
0925 12:17:34.855459 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.855461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.855780 (+ 319us) server_negotiation.cc:407] Connection header received
0925 12:17:34.856623 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.856624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.856634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.856666 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.857267 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.857962 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.859035 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.859258 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.859314 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.862374 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.862375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.862375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.862393 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.863924 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.864346 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.864347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.864347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.864411 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.864811 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.864812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.864812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.864940 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.865001 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.865796 (+ 795us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.865859 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:17:34.865676 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.855169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.3:48905)
0925 12:17:34.855796 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.855802 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.856523 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.856756 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.856757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.857103 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.857164 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.857166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.858068 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.858069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.858946 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.858948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.859408 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.859408 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.859468 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.859920 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.859925 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.861714 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.864255 (+ 2541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.864256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.864256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.864465 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.864711 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.864712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.864713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.864774 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.865394 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.865395 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.865433 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.865503 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.865575 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":42,"thread_start_us":86,"threads_started":1}
I20260925 12:17:34.866806 7727 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38175
I20260925 12:17:34.866891 7727 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:34.867048 7727 heartbeater.cc:511] Master 127.0.98.60:38175 requested a full tablet report, sending...
I20260925 12:17:34.867151 7728 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40821
I20260925 12:17:34.867194 7728 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:34.867256 7728 heartbeater.cc:511] Master 127.0.98.62:40821 requested a full tablet report, sending...
I20260925 12:17:34.867435 6490 ts_manager.cc:194] Re-registered known tserver with Master: ca1f60f9f9c040bdaf2f95beb2142646 (127.0.98.3:37931)
I20260925 12:17:34.867431 7274 ts_manager.cc:194] Registered new tserver with Master: ca1f60f9f9c040bdaf2f95beb2142646 (127.0.98.3:37931)
I20260925 12:17:34.867859 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48905
I20260925 12:17:34.869244 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.855168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43889 (local address 127.0.98.3:33509)
0925 12:17:34.855531 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.855565 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.856505 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.856738 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.856742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.857000 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:34.857152 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.857156 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.857796 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.857797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.858541 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.858543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.858916 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.858917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.859050 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.859522 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:34.859531 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:34.864280 (+ 4749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:34.866546 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.866547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.866548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.866748 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.868595 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:34.868595 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:34.868596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:34.868647 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:34.868913 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:34.868914 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:34.868951 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.869019 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.869118 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"spinlock_wait_cycles":6016,"thread_start_us":98,"threads_started":1}
I20260925 12:17:34.869604 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.855460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33509 (local address 127.0.98.61:43889)
0925 12:17:34.855502 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.855504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.855658 (+ 154us) server_negotiation.cc:407] Connection header received
0925 12:17:34.856609 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.856610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.856621 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.856678 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.857244 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.857702 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.858649 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.858827 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.858908 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.864383 (+ 5475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.864384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.864385 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.864391 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.866449 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.868468 (+ 2019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.868469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.868470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.868527 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.868705 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.868706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.868706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.868825 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.868876 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.869410 (+ 534us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.869480 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 12:17:34.869892 7729 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43889
I20260925 12:17:34.869943 7729 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:34.870012 7729 heartbeater.cc:511] Master 127.0.98.61:43889 requested a full tablet report, sending...
I20260925 12:17:34.870287 7089 ts_manager.cc:194] Registered new tserver with Master: ca1f60f9f9c040bdaf2f95beb2142646 (127.0.98.3:37931)
I20260925 12:17:34.873127 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.862954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40482 (local address 127.0.98.61:43889)
0925 12:17:34.863056 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.863058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.863189 (+ 131us) server_negotiation.cc:407] Connection header received
0925 12:17:34.863221 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.863221 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.863230 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.863277 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.864119 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.865410 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.867929 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.868215 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.868269 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.870147 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.870148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.870148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.870154 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.871883 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.872270 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.872271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.872271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.872318 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.872535 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.872536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.872536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.872641 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.872748 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.872946 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.873031 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":56,"threads_started":1}
I20260925 12:17:34.873515 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.862230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.0.98.60:38175)
0925 12:17:34.865589 (+ 3359us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.865591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.865597 (+ 6us) server_negotiation.cc:407] Connection header received
0925 12:17:34.865613 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.865614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.865623 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.865682 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.866828 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.867496 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.868482 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.868777 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.869067 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.870206 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.870207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.870207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.870213 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.871803 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.872155 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.872157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.872157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.872211 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.872466 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.872467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.872468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.872614 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.872717 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.873048 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.873119 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2357,"thread_start_us":51,"threads_started":1}
I20260925 12:17:34.874411 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.862412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47350 (local address 127.0.98.62:40821)
0925 12:17:34.864424 (+ 2012us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.864425 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.864431 (+ 6us) server_negotiation.cc:407] Connection header received
0925 12:17:34.864450 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.864450 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.864461 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.864526 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:34.867821 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.868580 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.870146 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.870474 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.870537 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.871866 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:34.871867 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:34.871867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:34.871873 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:34.873451 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.873837 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.873838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.873838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.873891 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:34.874054 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:34.874055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:34.874055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:34.874139 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:34.874201 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.874263 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.874314 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1987,"thread_start_us":86,"threads_started":1}
I20260925 12:17:34.875574 6490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:17:34.877025 6490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:17:34.877612 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b159ed78d924726b4d7a669d916a880" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: true } } }
I20260925 12:17:34.877700 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:34.878904 7745 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880: attempting to promote NON_VOTER b8ea31e1934c4a6da6f93396cbe910dc to VOTER
I20260925 12:17:34.879032 7745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: false } }
I20260925 12:17:34.879402 7284 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [term 1 LEARNER]: Refusing update from remote peer 8b159ed78d924726b4d7a669d916a880: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 12:17:34.879586 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 12:17:34.879751 7099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 1 FOLLOWER]: Refusing update from remote peer 8b159ed78d924726b4d7a669d916a880: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 12:17:34.880232 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 12:17:34.880385 7745 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b8ea31e1934c4a6da6f93396cbe910dc (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: false } } }
I20260925 12:17:34.880523 7284 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b8ea31e1934c4a6da6f93396cbe910dc (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: false } } }
I20260925 12:17:34.880990 7099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b8ea31e1934c4a6da6f93396cbe910dc (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: false } } }
I20260925 12:17:34.881474 7313 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b159ed78d924726b4d7a669d916a880" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: false } } }
I20260925 12:17:34.881538 7313 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:17:34.881527 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8b159ed78d924726b4d7a669d916a880" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: false } } }
I20260925 12:17:34.881544 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b159ed78d924726b4d7a669d916a880" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: false } } }
I20260925 12:17:34.881590 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:34.881623 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:34.881709 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b159ed78d924726b4d7a669d916a880" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: false } } }
I20260925 12:17:34.881767 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:34.881824 7238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b159ed78d924726b4d7a669d916a880" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: false } } }
I20260925 12:17:34.881897 7238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:17:34.885857 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.882625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40860 (local address 127.0.98.3:37931)
0925 12:17:34.882772 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.882777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.882786 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:17:34.882818 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.882820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.882833 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.883046 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:34.883086 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.883757 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.885053 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.885539 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.885628 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.885697 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.885760 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.885789 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260925 12:17:34.885859 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.882286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37931 (local address 127.0.0.1:40860)
0925 12:17:34.882340 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.882350 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.882397 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.882944 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.882945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.882947 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:34.882980 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.882982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.883850 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.883851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.884957 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.884959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.885627 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.885628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.885682 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.885685 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.885749 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.885785 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 12:17:34.886114 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.882339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39153 (local address 127.0.0.1:55564)
0925 12:17:34.882468 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.882476 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.882543 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.882984 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.882985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.882987 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:34.883031 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.883032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.883999 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.884000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.885047 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.885049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.885917 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.885918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.885978 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.885981 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.886021 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.886054 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260925 12:17:34.886408 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.882641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55564 (local address 127.0.98.2:39153)
0925 12:17:34.882816 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.882821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.882830 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:17:34.882867 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.882869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.882884 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.883107 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:34.883160 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.883920 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.885118 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.885829 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.886199 (+ 370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.886272 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.886295 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.886329 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":102,"threads_started":1}
I20260925 12:17:34.886499 7661 tablet_service.cc:1467] Processing CreateTablet for tablet b456a442c3294a788ded474fcb079bef (DEFAULT_TABLE table=test-table [id=e56c3abea3754428886256cee7a1c1ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:34.886812 7661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b456a442c3294a788ded474fcb079bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:34.887157 7525 tablet_service.cc:1467] Processing CreateTablet for tablet b456a442c3294a788ded474fcb079bef (DEFAULT_TABLE table=test-table [id=e56c3abea3754428886256cee7a1c1ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:34.887429 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b456a442c3294a788ded474fcb079bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:34.887701 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.882517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38055 (local address 127.0.0.1:57150)
0925 12:17:34.882616 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:34.882625 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:34.882671 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:34.883906 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:34.883907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:34.883909 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:34.883948 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.883949 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.885643 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.885644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.886624 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:34.886626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.887434 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.887435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:34.887490 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.887495 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:17:34.887546 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 12:17:34.887580 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":58,"threads_started":1}
I20260925 12:17:34.887918 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:34.882866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57150 (local address 127.0.98.1:38055)
0925 12:17:34.883736 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:34.883740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:34.883750 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:17:34.883786 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:34.883788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:34.883805 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:34.883893 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:34.884047 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.885550 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.886742 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:34.887348 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:34.887431 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:34.887491 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:34.887557 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 12:17:34.887589 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":75,"threads_started":1}
I20260925 12:17:34.888418 7389 tablet_service.cc:1467] Processing CreateTablet for tablet b456a442c3294a788ded474fcb079bef (DEFAULT_TABLE table=test-table [id=e56c3abea3754428886256cee7a1c1ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:34.888625 7389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b456a442c3294a788ded474fcb079bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:34.890139 7757 tablet_bootstrap.cc:492] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646: Bootstrap starting.
I20260925 12:17:34.890683 7758 tablet_bootstrap.cc:492] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2: Bootstrap starting.
I20260925 12:17:34.890733 7757 tablet_bootstrap.cc:654] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:34.891026 7757 log.cc:824] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:34.891273 7758 tablet_bootstrap.cc:654] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:34.891462 7759 tablet_bootstrap.cc:492] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77: Bootstrap starting.
I20260925 12:17:34.891706 7758 log.cc:824] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:34.891904 7757 tablet_bootstrap.cc:492] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646: No bootstrap required, opened a new log
I20260925 12:17:34.891975 7757 ts_tablet_manager.cc:1397] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:17:34.892048 7759 tablet_bootstrap.cc:654] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:34.892292 7759 log.cc:824] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:34.892573 7758 tablet_bootstrap.cc:492] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2: No bootstrap required, opened a new log
I20260925 12:17:34.892642 7758 ts_tablet_manager.cc:1397] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:17:34.893112 7759 tablet_bootstrap.cc:492] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77: No bootstrap required, opened a new log
I20260925 12:17:34.893699 7759 ts_tablet_manager.cc:1397] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:17:34.894655 7757 raft_consensus.cc:348] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 } } peers { permanent_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37931 } } peers { permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 } }
I20260925 12:17:34.894857 7757 raft_consensus.cc:374] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:34.894901 7757 raft_consensus.cc:729] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1f60f9f9c040bdaf2f95beb2142646, State: Initialized, Role: FOLLOWER
I20260925 12:17:34.894994 7757 consensus_queue.cc:260] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 } } peers { permanent_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37931 } } peers { permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 } }
I20260925 12:17:34.895013 7758 raft_consensus.cc:348] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 } } peers { permanent_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37931 } } peers { permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 } }
I20260925 12:17:34.895159 7758 raft_consensus.cc:374] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:34.895200 7758 raft_consensus.cc:729] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a05341f431ea4d129ca4e5d7afdbf1d2, State: Initialized, Role: FOLLOWER
I20260925 12:17:34.895211 7757 ts_tablet_manager.cc:1428] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 12:17:34.895304 7728 heartbeater.cc:503] Master 127.0.98.62:40821 was elected leader, sending a full tablet report...
I20260925 12:17:34.895438 7758 consensus_queue.cc:260] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 } } peers { permanent_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37931 } } peers { permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 } }
I20260925 12:17:34.895721 7758 ts_tablet_manager.cc:1428] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:17:34.895829 7592 heartbeater.cc:503] Master 127.0.98.62:40821 was elected leader, sending a full tablet report...
I20260925 12:17:34.896986 7759 raft_consensus.cc:348] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 } } peers { permanent_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37931 } } peers { permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 } }
I20260925 12:17:34.897118 7759 raft_consensus.cc:374] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:34.897161 7759 raft_consensus.cc:729] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fa57d079294d4589eb3e970a421c77, State: Initialized, Role: FOLLOWER
I20260925 12:17:34.897287 7759 consensus_queue.cc:260] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 } } peers { permanent_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37931 } } peers { permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 } }
I20260925 12:17:34.897524 7759 ts_tablet_manager.cc:1428] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 12:17:34.897557 7456 heartbeater.cc:503] Master 127.0.98.62:40821 was elected leader, sending a full tablet report...
W20260925 12:17:34.984287 7594 tablet.cc:2369] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:17:35.050634 7763 raft_consensus.cc:482] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:17:35.050729 7763 raft_consensus.cc:504] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 } } peers { permanent_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37931 } } peers { permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 } }
I20260925 12:17:35.051005 7763 leader_election.cc:288] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a05341f431ea4d129ca4e5d7afdbf1d2 (127.0.98.2:39153), 80fa57d079294d4589eb3e970a421c77 (127.0.98.1:38055)
I20260925 12:17:35.053725 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:35.051247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38067 (local address 127.0.98.2:39153)
0925 12:17:35.051294 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:35.051296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:35.051305 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:17:35.051369 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:35.051370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:35.051380 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:35.051456 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:35.051627 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:35.052123 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:35.053043 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:35.053396 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:35.053476 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:35.053487 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:35.053623 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 12:17:35.053657 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 12:17:35.053725 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:35.051166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39153 (local address 127.0.98.3:38067)
0925 12:17:35.051225 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:35.051238 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:35.051291 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:35.051458 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:35.051459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:35.051461 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:35.051542 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:35.051543 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:35.052201 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:35.052203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:35.052952 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:35.052954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:35.053474 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:35.053475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:35.053551 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:35.053554 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:35.053618 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 12:17:35.053657 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 12:17:35.053781 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:35.051331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48355 (local address 127.0.98.1:38055)
0925 12:17:35.051437 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:35.051438 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:35.051445 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:17:35.051465 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:35.051466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:35.051475 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:35.051533 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:35.051698 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:35.052194 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:35.053085 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:35.053438 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:35.053523 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:35.053533 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:35.053681 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 12:17:35.053716 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 12:17:35.053830 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:35.051274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38055 (local address 127.0.98.3:48355)
0925 12:17:35.051318 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:35.051327 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:35.051362 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:35.051535 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:35.051536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:35.051539 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:35.051600 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:35.051601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:35.052287 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:35.052288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:35.052992 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:35.052993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:35.053525 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:35.053526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:35.053600 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:35.053609 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 12:17:35.053674 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 12:17:35.053709 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 12:17:35.054162 7409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b456a442c3294a788ded474fcb079bef" candidate_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fa57d079294d4589eb3e970a421c77" is_pre_election: true
I20260925 12:17:35.054165 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b456a442c3294a788ded474fcb079bef" candidate_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" is_pre_election: true
I20260925 12:17:35.054306 7545 raft_consensus.cc:2454] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1f60f9f9c040bdaf2f95beb2142646 in term 0.
I20260925 12:17:35.054306 7409 raft_consensus.cc:2454] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1f60f9f9c040bdaf2f95beb2142646 in term 0.
I20260925 12:17:35.054525 7615 leader_election.cc:302] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a05341f431ea4d129ca4e5d7afdbf1d2, ca1f60f9f9c040bdaf2f95beb2142646; no voters:
I20260925 12:17:35.054661 7763 raft_consensus.cc:2781] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:17:35.054738 7763 raft_consensus.cc:482] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:17:35.054772 7763 raft_consensus.cc:3037] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:35.055405 7763 raft_consensus.cc:504] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 } } peers { permanent_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37931 } } peers { permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 } }
I20260925 12:17:35.055527 7763 leader_election.cc:288] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [CANDIDATE]: Term 1 election: Requested vote from peers a05341f431ea4d129ca4e5d7afdbf1d2 (127.0.98.2:39153), 80fa57d079294d4589eb3e970a421c77 (127.0.98.1:38055)
I20260925 12:17:35.055663 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b456a442c3294a788ded474fcb079bef" candidate_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2"
I20260925 12:17:35.055748 7545 raft_consensus.cc:3037] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:35.056015 7409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b456a442c3294a788ded474fcb079bef" candidate_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fa57d079294d4589eb3e970a421c77"
I20260925 12:17:35.056106 7409 raft_consensus.cc:3037] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:35.056636 7545 raft_consensus.cc:2454] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1f60f9f9c040bdaf2f95beb2142646 in term 1.
I20260925 12:17:35.056797 7615 leader_election.cc:302] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a05341f431ea4d129ca4e5d7afdbf1d2, ca1f60f9f9c040bdaf2f95beb2142646; no voters:
I20260925 12:17:35.056887 7763 raft_consensus.cc:2781] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:35.056986 7409 raft_consensus.cc:2454] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1f60f9f9c040bdaf2f95beb2142646 in term 1.
I20260925 12:17:35.057055 7763 raft_consensus.cc:686] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [term 1 LEADER]: Becoming Leader. State: Replica: ca1f60f9f9c040bdaf2f95beb2142646, State: Running, Role: LEADER
I20260925 12:17:35.057258 7763 consensus_queue.cc:237] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 } } peers { permanent_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37931 } } peers { permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 } }
I20260925 12:17:35.057986 6490 catalog_manager.cc:5495] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca1f60f9f9c040bdaf2f95beb2142646 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 } health_report { overall_health: UNKNOWN } } }
I20260925 12:17:35.066290 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:35.064059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40874 (local address 127.0.98.3:37931)
0925 12:17:35.064112 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:35.064114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:35.064131 (+ 17us) server_negotiation.cc:407] Connection header received
0925 12:17:35.064160 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:35.064161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:35.064177 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:35.064257 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:35.064402 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:35.064811 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:35.065554 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:35.065740 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:35.065832 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:35.066006 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:35.066094 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:35.066183 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:35.066205 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 12:17:35.066222 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 12:17:35.067776 7545 raft_consensus.cc:1261] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2 [term 1 FOLLOWER]: Refusing update from remote peer ca1f60f9f9c040bdaf2f95beb2142646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:35.067776 7409 raft_consensus.cc:1261] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77 [term 1 FOLLOWER]: Refusing update from remote peer ca1f60f9f9c040bdaf2f95beb2142646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:35.068048 7766 consensus_queue.cc:1035] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:35.068142 7763 consensus_queue.cc:1035] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:35.070606 7769 mvcc.cc:205] Tried to move back new op lower bound from 7333227131155378176 to 7333227131115126784. Current Snapshot: MvccSnapshot[applied={T|T < 7333227131155378176}]
I20260925 12:17:35.072444 7771 mvcc.cc:205] Tried to move back new op lower bound from 7333227131155378176 to 7333227131115126784. Current Snapshot: MvccSnapshot[applied={T|T < 7333227131155378176}]
I20260925 12:17:35.072695 7770 mvcc.cc:205] Tried to move back new op lower bound from 7333227131155378176 to 7333227131115126784. Current Snapshot: MvccSnapshot[applied={T|T < 7333227131155378176 or (T in {7333227131155378176})}]
I20260925 12:17:35.073488 6489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:35.073570 6489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:35.074641 6489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b159ed78d924726b4d7a669d916a880: Sending DeleteTablet for 3 replicas of tablet b456a442c3294a788ded474fcb079bef
I20260925 12:17:35.074959 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet b456a442c3294a788ded474fcb079bef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55564
I20260925 12:17:35.075001 7661 tablet_service.cc:1514] Processing DeleteTablet for tablet b456a442c3294a788ded474fcb079bef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40860
I20260925 12:17:35.075151 7389 tablet_service.cc:1514] Processing DeleteTablet for tablet b456a442c3294a788ded474fcb079bef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57150
I20260925 12:17:35.075260 7779 tablet_replica.cc:330] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646: stopping tablet replica
I20260925 12:17:35.075338 7779 raft_consensus.cc:2229] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:17:35.075439 7779 raft_consensus.cc:2258] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:35.075641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 6458
I20260925 12:17:35.076496 7779 ts_tablet_manager.cc:1905] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:35.078262 7779 ts_tablet_manager.cc:1918] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:17:35.078330 7779 log.cc:1196] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/b456a442c3294a788ded474fcb079bef
I20260925 12:17:35.078611 7779 ts_tablet_manager.cc:1939] T b456a442c3294a788ded474fcb079bef P ca1f60f9f9c040bdaf2f95beb2142646: Deleting consensus metadata
I20260925 12:17:35.078979 7780 tablet_replica.cc:330] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77: stopping tablet replica
I20260925 12:17:35.079072 7780 raft_consensus.cc:2229] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:35.079164 7780 raft_consensus.cc:2258] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:35.079294 7778 tablet_replica.cc:330] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2: stopping tablet replica
I20260925 12:17:35.079388 7778 raft_consensus.cc:2229] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:35.079476 7778 raft_consensus.cc:2258] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:35.079885 7780 ts_tablet_manager.cc:1905] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:35.079896 7778 ts_tablet_manager.cc:1905] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:35.081372 7780 ts_tablet_manager.cc:1918] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:17:35.081480 7780 log.cc:1196] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-0/wal/wals/b456a442c3294a788ded474fcb079bef
I20260925 12:17:35.081591 7778 ts_tablet_manager.cc:1918] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:17:35.081657 7778 log.cc:1196] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/b456a442c3294a788ded474fcb079bef
I20260925 12:17:35.081836 7780 ts_tablet_manager.cc:1939] T b456a442c3294a788ded474fcb079bef P 80fa57d079294d4589eb3e970a421c77: Deleting consensus metadata
I20260925 12:17:35.081878 7778 ts_tablet_manager.cc:1939] T b456a442c3294a788ded474fcb079bef P a05341f431ea4d129ca4e5d7afdbf1d2: Deleting consensus metadata
W20260925 12:17:35.082460 7480 connection.cc:535] server connection from 127.0.0.1:55564 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 12:17:35.082597 7614 connection.cc:535] server connection from 127.0.0.1:40860 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 12:17:35.082625 7261 connection.cc:535] server connection from 127.0.0.1:54434 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 12:17:35.089735 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:35.083970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40484 (local address 127.0.98.61:43889)
0925 12:17:35.084076 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:35.084085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:35.084093 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:35.084113 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:35.084114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:35.084124 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:35.084172 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:35.084721 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:35.085179 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:35.085863 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:35.086088 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:35.086162 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:35.087391 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:35.087392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:35.087392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:35.087400 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:35.088679 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:35.089016 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:35.089017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:35.089018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:35.089071 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:35.089259 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:35.089260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:35.089260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:35.089361 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:35.089463 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:35.089560 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 12:17:35.089633 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 12:17:35.095955 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:35.084067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54452 (local address 127.0.98.60:38175)
0925 12:17:35.084132 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:35.084134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:35.089747 (+ 5613us) server_negotiation.cc:407] Connection header received
0925 12:17:35.089845 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:35.089845 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:35.089858 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:35.089885 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:35.090477 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:35.091195 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:35.091871 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:35.092206 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:35.092275 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:35.093352 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:35.093353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:35.093354 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:35.093361 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:35.094929 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:35.095219 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:35.095219 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:35.095220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:35.095277 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:35.095449 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:35.095450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:35.095450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:35.095572 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:35.095689 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:35.095793 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 12:17:35.095869 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 12:17:36.072548 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.072276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.2:40295)
0925 12:17:36.072395 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.072489 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
W20260925 12:17:36.073032 7592 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40821 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
I20260925 12:17:36.073140 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.072912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.3:39233)
0925 12:17:36.073025 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.073092 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":53,"threads_started":1}
W20260925 12:17:36.073302 7728 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40821 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
I20260925 12:17:36.073392 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.073059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.1:35331)
0925 12:17:36.073272 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.073336 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":53,"threads_started":1}
W20260925 12:17:36.073546 7456 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40821 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
I20260925 12:17:36.575488 7790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b159ed78d924726b4d7a669d916a880)
I20260925 12:17:36.575603 7790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: false } }
I20260925 12:17:36.575896 7790 leader_election.cc:288] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b159ed78d924726b4d7a669d916a880 (127.0.98.62:40821), b8ea31e1934c4a6da6f93396cbe910dc (127.0.98.60:38175)
I20260925 12:17:36.576184 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.575971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.0.1:47810)
0925 12:17:36.576092 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.576144 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":45,"threads_started":1}
W20260925 12:17:36.576300 7073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
I20260925 12:17:36.576545 7795 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b159ed78d924726b4d7a669d916a880)
I20260925 12:17:36.576678 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.576601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.0.1:47816)
0925 12:17:36.576635 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.576655 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 12:17:36.576665 7795 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: false } }
W20260925 12:17:36.576749 7073 leader_election.cc:334] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b159ed78d924726b4d7a669d916a880 (127.0.98.62:40821): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
I20260925 12:17:36.576958 7795 leader_election.cc:288] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b159ed78d924726b4d7a669d916a880 (127.0.98.62:40821), 11fca8bf24634bee88f5e4324d199660 (127.0.98.61:43889)
I20260925 12:17:36.577260 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.577017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.0.1:47824)
0925 12:17:36.577182 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.577225 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":52,"threads_started":1}
W20260925 12:17:36.577415 7258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
I20260925 12:17:36.577800 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.577708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.0.1:47832)
0925 12:17:36.577743 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.577772 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260925 12:17:36.577883 7258 leader_election.cc:334] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b159ed78d924726b4d7a669d916a880 (127.0.98.62:40821): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
I20260925 12:17:36.579489 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.576175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54456 (local address 127.0.98.60:38175)
0925 12:17:36.576281 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:36.576283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:36.576292 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:17:36.576317 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:36.576319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:36.576330 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:36.576393 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:36.576592 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.577368 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.578719 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.579149 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.579230 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:36.579241 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:36.579400 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 12:17:36.579436 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":49,"threads_started":1}
I20260925 12:17:36.579493 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.576052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38175 (local address 127.0.0.1:54456)
0925 12:17:36.576160 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.576169 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:36.576218 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:36.576425 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:36.576426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:36.576429 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:36.576492 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:36.576494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.577432 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.577433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:36.578638 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:36.578640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.579235 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.579236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:36.579308 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:36.579314 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 12:17:36.579393 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 12:17:36.579433 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":46,"threads_started":1}
I20260925 12:17:36.579797 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "11fca8bf24634bee88f5e4324d199660" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" is_pre_election: true
I20260925 12:17:36.579908 7284 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11fca8bf24634bee88f5e4324d199660 in term 1.
I20260925 12:17:36.580047 7074 leader_election.cc:302] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 11fca8bf24634bee88f5e4324d199660, b8ea31e1934c4a6da6f93396cbe910dc; no voters: 8b159ed78d924726b4d7a669d916a880
I20260925 12:17:36.580135 7790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 12:17:36.580173 7790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8b159ed78d924726b4d7a669d916a880)
I20260925 12:17:36.580195 7790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 1 FOLLOWER]: Advancing to term 2
I20260925 12:17:36.580788 7790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: false } }
I20260925 12:17:36.580912 7790 leader_election.cc:288] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [CANDIDATE]: Term 2 election: Requested vote from peers 8b159ed78d924726b4d7a669d916a880 (127.0.98.62:40821), b8ea31e1934c4a6da6f93396cbe910dc (127.0.98.60:38175)
I20260925 12:17:36.581079 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.580974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.0.1:47836)
0925 12:17:36.581006 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.581054 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 12:17:36.581137 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "11fca8bf24634bee88f5e4324d199660" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b8ea31e1934c4a6da6f93396cbe910dc"
I20260925 12:17:36.581208 7284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [term 1 FOLLOWER]: Advancing to term 2
I20260925 12:17:36.581382 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.577093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43889 (local address 127.0.0.1:40494)
0925 12:17:36.577241 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.577250 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:36.577305 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:36.577694 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:36.577695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:36.577697 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:36.577736 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:36.577737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.578545 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.578545 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:36.579664 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:36.579667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.581182 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.581182 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:36.581245 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:36.581249 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:17:36.581299 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 12:17:36.581333 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":46,"threads_started":1}
I20260925 12:17:36.581552 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.577123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40494 (local address 127.0.98.61:43889)
0925 12:17:36.577237 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:36.577239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:36.577291 (+ 52us) server_negotiation.cc:407] Connection header received
0925 12:17:36.577611 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:36.577611 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:36.577621 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:36.577680 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:36.577788 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.578472 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.579725 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.581124 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.581182 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:36.581192 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:36.581446 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 12:17:36.581500 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":73,"threads_started":1}
I20260925 12:17:36.581943 7284 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11fca8bf24634bee88f5e4324d199660 in term 2.
I20260925 12:17:36.582111 7074 leader_election.cc:302] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11fca8bf24634bee88f5e4324d199660, b8ea31e1934c4a6da6f93396cbe910dc; no voters:
I20260925 12:17:36.582191 7790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 12:17:36.582190 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "11fca8bf24634bee88f5e4324d199660" is_pre_election: true
I20260925 12:17:36.582345 7790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [term 2 LEADER]: Becoming Leader. State: Replica: 11fca8bf24634bee88f5e4324d199660, State: Running, Role: LEADER
I20260925 12:17:36.582424 7790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: false } }
I20260925 12:17:36.582458 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.582334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.0.1:47846)
0925 12:17:36.582378 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.582418 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 12:17:36.582523 7073 leader_election.cc:334] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8b159ed78d924726b4d7a669d916a880 (127.0.98.62:40821): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
I20260925 12:17:36.582904 7260 leader_election.cc:302] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8ea31e1934c4a6da6f93396cbe910dc; no voters: 11fca8bf24634bee88f5e4324d199660, 8b159ed78d924726b4d7a669d916a880
I20260925 12:17:36.583024 7795 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:17:36.583037 7800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11fca8bf24634bee88f5e4324d199660. Latest consensus state: current_term: 2 leader_uuid: "11fca8bf24634bee88f5e4324d199660" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: false } } }
I20260925 12:17:36.583122 7800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:36.583269 7802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:36.583753 7802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:17:36.583833 7802 catalog_manager.cc:1259] Loaded cluster ID: 9ebbedcdd3ba4f3ca9c70ee49c9b30d5
I20260925 12:17:36.583865 7802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:17:36.584043 7802 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:36.584131 7802 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: Loaded TSK: 0
I20260925 12:17:36.584273 7802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:17:36.625901 7088 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:17:36.627359 7088 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:17:36.628222 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.628106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.0.1:47868)
0925 12:17:36.628170 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.628198 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 12:17:36.628233 7284 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [term 2 FOLLOWER]: Refusing update from remote peer 11fca8bf24634bee88f5e4324d199660: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 12:17:36.628473 7800 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 12:17:36.628639 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.628535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.0.1:47876)
0925 12:17:36.628582 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.628609 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 12:17:36.628691 7073 consensus_peers.cc:487] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 -> Peer 8b159ed78d924726b4d7a669d916a880 (127.0.98.62:40821): Couldn't send request to peer 8b159ed78d924726b4d7a669d916a880. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 12:17:36.629074 7795 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11fca8bf24634bee88f5e4324d199660. Latest consensus state: current_term: 2 leader_uuid: "11fca8bf24634bee88f5e4324d199660" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: false } } }
I20260925 12:17:36.629168 7795 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:17:36.629542 7803 mvcc.cc:205] Tried to move back new op lower bound from 7333227137547440128 to 7333227137362092032. Current Snapshot: MvccSnapshot[applied={T|T < 7333227137547440128}]
I20260925 12:17:36.629609 7790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "11fca8bf24634bee88f5e4324d199660" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: false } } }
I20260925 12:17:36.629694 7790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:36.629822 7795 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "11fca8bf24634bee88f5e4324d199660" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b159ed78d924726b4d7a669d916a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40821 } } peers { permanent_uuid: "11fca8bf24634bee88f5e4324d199660" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43889 } attrs { promote: false } } peers { permanent_uuid: "b8ea31e1934c4a6da6f93396cbe910dc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38175 } attrs { promote: false } } }
I20260925 12:17:36.629892 7795 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ea31e1934c4a6da6f93396cbe910dc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:17:36.634436 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.631369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37931 (local address 127.0.0.1:40880)
0925 12:17:36.631395 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.631404 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:36.631437 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:36.631720 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:36.631721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:36.631723 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:36.631760 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:36.631761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.632445 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.632446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:36.633297 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:36.633299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.634254 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.634255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:36.634306 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:36.634309 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:36.634352 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 12:17:36.634383 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260925 12:17:36.634447 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.631449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40880 (local address 127.0.98.3:37931)
0925 12:17:36.631573 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:36.631575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:36.631581 (+ 6us) server_negotiation.cc:407] Connection header received
0925 12:17:36.631603 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:36.631603 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:36.631614 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:36.631812 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:36.631855 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.632373 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.633403 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.634175 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.634256 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:36.634266 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:36.634357 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 12:17:36.634386 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":88,"threads_started":1}
I20260925 12:17:36.634589 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.631441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38055 (local address 127.0.0.1:57164)
0925 12:17:36.631479 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.631520 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:36.631576 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:36.631985 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:36.631986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:36.631989 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:36.632042 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:36.632043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.632973 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.632974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:36.633696 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:36.633697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.634405 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.634406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:36.634471 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:36.634474 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:36.634499 (+ 25us) client_negotiation.cc:240] Negotiation successful
0925 12:17:36.634532 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260925 12:17:36.634675 7661 tablet_service.cc:1467] Processing CreateTablet for tablet 23c46827fe684638986b9f7de1616d5c (DEFAULT_TABLE table=test-table [id=11239bde3edf4085b0ad9e3c6400f2a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:36.634792 7661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c46827fe684638986b9f7de1616d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:36.634907 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.631712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57164 (local address 127.0.98.1:38055)
0925 12:17:36.631856 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:36.631858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:36.631866 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:36.631889 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:36.631890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:36.631899 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:36.632102 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:36.632139 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.632864 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.633817 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.634320 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.634733 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:36.634746 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:36.634779 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 12:17:36.634812 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":47,"threads_started":1}
I20260925 12:17:36.635004 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.631562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39153 (local address 127.0.0.1:55568)
0925 12:17:36.631679 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.631688 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:36.631724 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:36.632074 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:36.632075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:36.632077 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:36.632106 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:36.632107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.632950 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.632950 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:36.634067 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:36.634068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.634749 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.634750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:36.634837 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:36.634841 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:17:36.634895 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 12:17:36.634932 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":26,"thread_start_us":81,"threads_started":1}
I20260925 12:17:36.635092 7389 tablet_service.cc:1467] Processing CreateTablet for tablet 23c46827fe684638986b9f7de1616d5c (DEFAULT_TABLE table=test-table [id=11239bde3edf4085b0ad9e3c6400f2a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:36.635214 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.631759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55568 (local address 127.0.98.2:39153)
0925 12:17:36.631943 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:36.631945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:36.631953 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:36.631974 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:36.631975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:36.631983 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:36.632035 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:36.632175 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.632873 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.634139 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.634646 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.634708 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:36.634722 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:36.635078 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 12:17:36.635116 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":45,"threads_started":1}
I20260925 12:17:36.635228 7389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c46827fe684638986b9f7de1616d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:36.636080 7812 tablet_bootstrap.cc:492] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646: Bootstrap starting.
I20260925 12:17:36.636453 7813 tablet_bootstrap.cc:492] T 23c46827fe684638986b9f7de1616d5c P 80fa57d079294d4589eb3e970a421c77: Bootstrap starting.
I20260925 12:17:36.636544 7812 tablet_bootstrap.cc:654] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:36.636652 7525 tablet_service.cc:1467] Processing CreateTablet for tablet 23c46827fe684638986b9f7de1616d5c (DEFAULT_TABLE table=test-table [id=11239bde3edf4085b0ad9e3c6400f2a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:36.636778 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c46827fe684638986b9f7de1616d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:36.636847 7813 tablet_bootstrap.cc:654] T 23c46827fe684638986b9f7de1616d5c P 80fa57d079294d4589eb3e970a421c77: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:36.637096 7812 tablet_bootstrap.cc:492] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646: No bootstrap required, opened a new log
I20260925 12:17:36.637172 7812 ts_tablet_manager.cc:1397] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:36.637280 7812 raft_consensus.cc:348] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37931 } } peers { permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 } } peers { permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 } }
I20260925 12:17:36.637365 7812 raft_consensus.cc:374] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:36.637375 7813 tablet_bootstrap.cc:492] T 23c46827fe684638986b9f7de1616d5c P 80fa57d079294d4589eb3e970a421c77: No bootstrap required, opened a new log
I20260925 12:17:36.637387 7812 raft_consensus.cc:729] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1f60f9f9c040bdaf2f95beb2142646, State: Initialized, Role: FOLLOWER
I20260925 12:17:36.637440 7813 ts_tablet_manager.cc:1397] T 23c46827fe684638986b9f7de1616d5c P 80fa57d079294d4589eb3e970a421c77: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:36.637463 7812 consensus_queue.cc:260] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37931 } } peers { permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 } } peers { permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 } }
I20260925 12:17:36.637552 7813 raft_consensus.cc:348] T 23c46827fe684638986b9f7de1616d5c P 80fa57d079294d4589eb3e970a421c77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37931 } } peers { permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 } } peers { permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 } }
I20260925 12:17:36.637619 7813 raft_consensus.cc:374] T 23c46827fe684638986b9f7de1616d5c P 80fa57d079294d4589eb3e970a421c77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:36.637640 7813 raft_consensus.cc:729] T 23c46827fe684638986b9f7de1616d5c P 80fa57d079294d4589eb3e970a421c77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fa57d079294d4589eb3e970a421c77, State: Initialized, Role: FOLLOWER
I20260925 12:17:36.637681 7813 consensus_queue.cc:260] T 23c46827fe684638986b9f7de1616d5c P 80fa57d079294d4589eb3e970a421c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37931 } } peers { permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 } } peers { permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 } }
I20260925 12:17:36.637854 7813 ts_tablet_manager.cc:1428] T 23c46827fe684638986b9f7de1616d5c P 80fa57d079294d4589eb3e970a421c77: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:36.637590 7812 ts_tablet_manager.cc:1428] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:36.638271 7729 heartbeater.cc:503] Master 127.0.98.61:43889 was elected leader, sending a full tablet report...
I20260925 12:17:36.638676 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.637827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.3:59219)
0925 12:17:36.638498 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.638644 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":66,"threads_started":1}
I20260925 12:17:36.638708 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.638355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.1:46021)
0925 12:17:36.638651 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.638679 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":62,"threads_started":1}
I20260925 12:17:36.638854 7819 tablet_bootstrap.cc:492] T 23c46827fe684638986b9f7de1616d5c P a05341f431ea4d129ca4e5d7afdbf1d2: Bootstrap starting.
I20260925 12:17:36.638970 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.638863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.3:38229)
0925 12:17:36.638912 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.638942 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 12:17:36.639191 7819 tablet_bootstrap.cc:654] T 23c46827fe684638986b9f7de1616d5c P a05341f431ea4d129ca4e5d7afdbf1d2: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:36.639534 7458 heartbeater.cc:503] Master 127.0.98.61:43889 was elected leader, sending a full tablet report...
I20260925 12:17:36.639899 7819 tablet_bootstrap.cc:492] T 23c46827fe684638986b9f7de1616d5c P a05341f431ea4d129ca4e5d7afdbf1d2: No bootstrap required, opened a new log
I20260925 12:17:36.639938 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.639599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.1:45545)
0925 12:17:36.639806 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.639834 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260925 12:17:36.639952 7819 ts_tablet_manager.cc:1397] T 23c46827fe684638986b9f7de1616d5c P a05341f431ea4d129ca4e5d7afdbf1d2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:36.640065 7819 raft_consensus.cc:348] T 23c46827fe684638986b9f7de1616d5c P a05341f431ea4d129ca4e5d7afdbf1d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37931 } } peers { permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 } } peers { permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 } }
I20260925 12:17:36.640125 7819 raft_consensus.cc:374] T 23c46827fe684638986b9f7de1616d5c P a05341f431ea4d129ca4e5d7afdbf1d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:36.640185 7819 raft_consensus.cc:729] T 23c46827fe684638986b9f7de1616d5c P a05341f431ea4d129ca4e5d7afdbf1d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a05341f431ea4d129ca4e5d7afdbf1d2, State: Initialized, Role: FOLLOWER
I20260925 12:17:36.640239 7819 consensus_queue.cc:260] T 23c46827fe684638986b9f7de1616d5c P a05341f431ea4d129ca4e5d7afdbf1d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37931 } } peers { permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 } } peers { permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 } }
I20260925 12:17:36.640413 7819 ts_tablet_manager.cc:1428] T 23c46827fe684638986b9f7de1616d5c P a05341f431ea4d129ca4e5d7afdbf1d2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:36.640818 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.640555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.2:45883)
0925 12:17:36.640749 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.640782 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":71,"threads_started":1}
I20260925 12:17:36.641283 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.641163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.2:52751)
0925 12:17:36.641210 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.641233 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 12:17:36.856231 7730 tablet.cc:2369] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 12:17:36.865955 7459 tablet.cc:2369] T 23c46827fe684638986b9f7de1616d5c P 80fa57d079294d4589eb3e970a421c77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:17:36.922380 7816 raft_consensus.cc:482] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:17:36.922456 7816 raft_consensus.cc:504] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37931 } } peers { permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 } } peers { permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 } }
I20260925 12:17:36.922601 7816 leader_election.cc:288] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fa57d079294d4589eb3e970a421c77 (127.0.98.1:38055), a05341f431ea4d129ca4e5d7afdbf1d2 (127.0.98.2:39153)
I20260925 12:17:36.922827 7409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c46827fe684638986b9f7de1616d5c" candidate_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fa57d079294d4589eb3e970a421c77" is_pre_election: true
I20260925 12:17:36.922873 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c46827fe684638986b9f7de1616d5c" candidate_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" is_pre_election: true
I20260925 12:17:36.922941 7409 raft_consensus.cc:2454] T 23c46827fe684638986b9f7de1616d5c P 80fa57d079294d4589eb3e970a421c77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1f60f9f9c040bdaf2f95beb2142646 in term 0.
I20260925 12:17:36.922945 7545 raft_consensus.cc:2454] T 23c46827fe684638986b9f7de1616d5c P a05341f431ea4d129ca4e5d7afdbf1d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1f60f9f9c040bdaf2f95beb2142646 in term 0.
I20260925 12:17:36.923095 7615 leader_election.cc:302] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fa57d079294d4589eb3e970a421c77, ca1f60f9f9c040bdaf2f95beb2142646; no voters:
I20260925 12:17:36.923202 7816 raft_consensus.cc:2781] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:17:36.923244 7816 raft_consensus.cc:482] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:17:36.923264 7816 raft_consensus.cc:3037] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:36.924090 7816 raft_consensus.cc:504] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37931 } } peers { permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 } } peers { permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 } }
I20260925 12:17:36.924213 7816 leader_election.cc:288] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [CANDIDATE]: Term 1 election: Requested vote from peers 80fa57d079294d4589eb3e970a421c77 (127.0.98.1:38055), a05341f431ea4d129ca4e5d7afdbf1d2 (127.0.98.2:39153)
I20260925 12:17:36.924355 7409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c46827fe684638986b9f7de1616d5c" candidate_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fa57d079294d4589eb3e970a421c77"
I20260925 12:17:36.924440 7409 raft_consensus.cc:3037] T 23c46827fe684638986b9f7de1616d5c P 80fa57d079294d4589eb3e970a421c77 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:36.924400 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c46827fe684638986b9f7de1616d5c" candidate_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2"
I20260925 12:17:36.924501 7545 raft_consensus.cc:3037] T 23c46827fe684638986b9f7de1616d5c P a05341f431ea4d129ca4e5d7afdbf1d2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:36.925017 7409 raft_consensus.cc:2454] T 23c46827fe684638986b9f7de1616d5c P 80fa57d079294d4589eb3e970a421c77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1f60f9f9c040bdaf2f95beb2142646 in term 1.
I20260925 12:17:36.925149 7545 raft_consensus.cc:2454] T 23c46827fe684638986b9f7de1616d5c P a05341f431ea4d129ca4e5d7afdbf1d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1f60f9f9c040bdaf2f95beb2142646 in term 1.
I20260925 12:17:36.925168 7615 leader_election.cc:302] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fa57d079294d4589eb3e970a421c77, ca1f60f9f9c040bdaf2f95beb2142646; no voters:
I20260925 12:17:36.925271 7816 raft_consensus.cc:2781] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:36.925418 7816 raft_consensus.cc:686] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [term 1 LEADER]: Becoming Leader. State: Replica: ca1f60f9f9c040bdaf2f95beb2142646, State: Running, Role: LEADER
I20260925 12:17:36.925472 7816 consensus_queue.cc:237] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37931 } } peers { permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 } } peers { permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 } }
I20260925 12:17:36.925781 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.925661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.3:44759)
0925 12:17:36.925718 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.925749 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 12:17:36.926051 7089 catalog_manager.cc:5495] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca1f60f9f9c040bdaf2f95beb2142646 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1f60f9f9c040bdaf2f95beb2142646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 } health_report { overall_health: UNKNOWN } } }
I20260925 12:17:36.985549 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.983219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40884 (local address 127.0.98.3:37931)
0925 12:17:36.983284 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:36.983286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:36.983296 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:17:36.983322 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:36.983322 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:36.983351 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:36.983400 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:36.983527 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.984133 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.984839 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:36.985056 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:36.985125 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:36.985271 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:36.985302 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:36.985353 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:36.985439 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 12:17:36.985459 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 12:17:36.986478 7545 raft_consensus.cc:1261] T 23c46827fe684638986b9f7de1616d5c P a05341f431ea4d129ca4e5d7afdbf1d2 [term 1 FOLLOWER]: Refusing update from remote peer ca1f60f9f9c040bdaf2f95beb2142646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:36.986533 7409 raft_consensus.cc:1261] T 23c46827fe684638986b9f7de1616d5c P 80fa57d079294d4589eb3e970a421c77 [term 1 FOLLOWER]: Refusing update from remote peer ca1f60f9f9c040bdaf2f95beb2142646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:36.986712 7816 consensus_queue.cc:1035] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a05341f431ea4d129ca4e5d7afdbf1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:36.986881 7824 consensus_queue.cc:1035] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fa57d079294d4589eb3e970a421c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:36.987108 7593 heartbeater.cc:503] Master 127.0.98.61:43889 was elected leader, sending a full tablet report...
I20260925 12:17:36.987459 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.987335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.2:36035)
0925 12:17:36.987395 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.987427 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 12:17:36.987507 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.987422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.1:51241)
0925 12:17:36.987458 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.987478 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 12:17:36.988559 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.988467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.3:37011)
0925 12:17:36.988502 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.988527 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 12:17:36.988683 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.988573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.1:58135)
0925 12:17:36.988638 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.988659 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 12:17:36.988814 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.988744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.3:57401)
0925 12:17:36.988775 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.988793 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 12:17:36.989159 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.988619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40821 (local address 127.0.98.2:38293)
0925 12:17:36.989097 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.989126 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461}
I20260925 12:17:36.989938 7089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:36.990043 7089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:36.991252 7089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11fca8bf24634bee88f5e4324d199660: Sending DeleteTablet for 3 replicas of tablet 23c46827fe684638986b9f7de1616d5c
I20260925 12:17:36.991535 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet 23c46827fe684638986b9f7de1616d5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:36 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:55568
I20260925 12:17:36.991540 7661 tablet_service.cc:1514] Processing DeleteTablet for tablet 23c46827fe684638986b9f7de1616d5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:36 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40880
I20260925 12:17:36.991570 7389 tablet_service.cc:1514] Processing DeleteTablet for tablet 23c46827fe684638986b9f7de1616d5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:36 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:57164
I20260925 12:17:36.991740 7837 tablet_replica.cc:330] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646: stopping tablet replica
I20260925 12:17:36.991791 7837 raft_consensus.cc:2229] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:17:36.991851 7837 raft_consensus.cc:2258] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:36.991875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 7326
I20260925 12:17:36.992859 7837 ts_tablet_manager.cc:1905] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:36.991775 7836 tablet_replica.cc:330] T 23c46827fe684638986b9f7de1616d5c P a05341f431ea4d129ca4e5d7afdbf1d2: stopping tablet replica
I20260925 12:17:36.993741 7836 raft_consensus.cc:2229] T 23c46827fe684638986b9f7de1616d5c P a05341f431ea4d129ca4e5d7afdbf1d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:36.993791 7836 raft_consensus.cc:2258] T 23c46827fe684638986b9f7de1616d5c P a05341f431ea4d129ca4e5d7afdbf1d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:36.994081 7836 ts_tablet_manager.cc:1905] T 23c46827fe684638986b9f7de1616d5c P a05341f431ea4d129ca4e5d7afdbf1d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:36.994226 7837 ts_tablet_manager.cc:1918] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:17:36.994285 7837 log.cc:1196] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/23c46827fe684638986b9f7de1616d5c
I20260925 12:17:36.994505 7837 ts_tablet_manager.cc:1939] T 23c46827fe684638986b9f7de1616d5c P ca1f60f9f9c040bdaf2f95beb2142646: Deleting consensus metadata
I20260925 12:17:36.994781 7073 catalog_manager.cc:4841] TS ca1f60f9f9c040bdaf2f95beb2142646 (127.0.98.3:37931): tablet 23c46827fe684638986b9f7de1616d5c (table test-table [id=11239bde3edf4085b0ad9e3c6400f2a9]) successfully deleted
I20260925 12:17:36.995365 7836 ts_tablet_manager.cc:1918] T 23c46827fe684638986b9f7de1616d5c P a05341f431ea4d129ca4e5d7afdbf1d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:17:36.995441 7836 log.cc:1196] T 23c46827fe684638986b9f7de1616d5c P a05341f431ea4d129ca4e5d7afdbf1d2: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/23c46827fe684638986b9f7de1616d5c
I20260925 12:17:36.995673 7836 ts_tablet_manager.cc:1939] T 23c46827fe684638986b9f7de1616d5c P a05341f431ea4d129ca4e5d7afdbf1d2: Deleting consensus metadata
I20260925 12:17:36.995935 7074 catalog_manager.cc:4841] TS a05341f431ea4d129ca4e5d7afdbf1d2 (127.0.98.2:39153): tablet 23c46827fe684638986b9f7de1616d5c (table test-table [id=11239bde3edf4085b0ad9e3c6400f2a9]) successfully deleted
I20260925 12:17:36.998855 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:36.998734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38055 (local address 127.0.0.1:57174)
0925 12:17:36.998784 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:36.998826 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 12:17:36.998871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 7462
W20260925 12:17:36.998994 7074 catalog_manager.cc:4584] TS 80fa57d079294d4589eb3e970a421c77 (127.0.98.1:38055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 23c46827fe684638986b9f7de1616d5c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38055: connect: Connection refused (error 111)
I20260925 12:17:37.003485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 7598
I20260925 12:17:37.008510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 6954
I20260925 12:17:37.012544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 7143
2026-09-25T12:17:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4740 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:17:37 dist-test-slave-clmx krb5kdc[7841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:17:37 dist-test-slave-clmx krb5kdc[7841](info): set up 2 sockets
Sep 25 12:17:37 dist-test-slave-clmx krb5kdc[7841](info): commencing operation
krb5kdc: starting...
W20260925 12:17:38.007994 392 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:17:38 dist-test-slave-clmx krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:17:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260925 12:17:38.110981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34333
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:38.185782 7854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:38.185959 7854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:38.185979 7854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:38.187026 7854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:38.187055 7854 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 12:17:38.187067 7854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:38.187079 7854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:38.187091 7854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:38.187103 7854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:17:38.188390 7854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34333
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:38.188549 7854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:38.188687 7854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:38.192972 7859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:38.193015 7854 server_base.cc:1034] running on GCE node
W20260925 12:17:38.192981 7860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:38.192983 7862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:38.193360 7854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:38.193630 7854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:38.194787 7854 hybrid_clock.cc:648] HybridClock initialized: now 1790338658194778 us; error 36 us; skew 500 ppm
Sep 25 12:17:38 dist-test-slave-clmx krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338658, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:38.196868 7854 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260925 12:17:38.197177 7854 webserver.cc:460] Webserver started at http://127.0.98.62:40357/ using document root <none> and password file <none>
I20260925 12:17:38.197324 7854 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:38.197361 7854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:38.197432 7854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:38.198246 7854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "3f84eb8c520a4b528af0e9fe52a57411"
format_stamp: "Formatted at 2026-09-25 12:17:38 on dist-test-slave-clmx"
server_key: "c8d5cec52bc9e61c9edd338a5ad11839"
server_key_iv: "5109c95ec3c62c8063d749536f7d3e33"
server_key_version: "encryptionkey@0"
I20260925 12:17:38.198506 7854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "3f84eb8c520a4b528af0e9fe52a57411"
format_stamp: "Formatted at 2026-09-25 12:17:38 on dist-test-slave-clmx"
server_key: "c8d5cec52bc9e61c9edd338a5ad11839"
server_key_iv: "5109c95ec3c62c8063d749536f7d3e33"
server_key_version: "encryptionkey@0"
I20260925 12:17:38.199594 7854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:17:38.200213 7869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:38.200345 7854 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:38.200433 7854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "3f84eb8c520a4b528af0e9fe52a57411"
format_stamp: "Formatted at 2026-09-25 12:17:38 on dist-test-slave-clmx"
server_key: "c8d5cec52bc9e61c9edd338a5ad11839"
server_key_iv: "5109c95ec3c62c8063d749536f7d3e33"
server_key_version: "encryptionkey@0"
I20260925 12:17:38.200479 7854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:38.212711 7854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:38.214250 7854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:38.214341 7854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:38.218201 7854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34333
I20260925 12:17:38.218269 7931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34333 every 8 connection(s)
I20260925 12:17:38.218570 7854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:38.219149 7932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:38.221025 7932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411: Bootstrap starting.
I20260925 12:17:38.221480 7932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:38.221692 7932 log.cc:824] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:38.222235 7932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411: No bootstrap required, opened a new log
I20260925 12:17:38.224241 7932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f84eb8c520a4b528af0e9fe52a57411" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34333 } }
I20260925 12:17:38.224437 7932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:38.224443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 7854
I20260925 12:17:38.224488 7932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f84eb8c520a4b528af0e9fe52a57411, State: Initialized, Role: FOLLOWER
I20260925 12:17:38.224545 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:17:38.224632 7932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f84eb8c520a4b528af0e9fe52a57411" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34333 } }
I20260925 12:17:38.224709 7932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:38.224726 392 external_mini_cluster.cc:1428] Setting key e2ffe4ef01e3cc36b4f719a070fb3213
I20260925 12:17:38.224738 7932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:38.224781 7932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:38.225472 7932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f84eb8c520a4b528af0e9fe52a57411" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34333 } }
I20260925 12:17:38.225584 7932 leader_election.cc:302] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f84eb8c520a4b528af0e9fe52a57411; no voters:
I20260925 12:17:38.225723 7932 leader_election.cc:288] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:17:38.225900 7937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:38.225965 7932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:38.226012 7937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411 [term 1 LEADER]: Becoming Leader. State: Replica: 3f84eb8c520a4b528af0e9fe52a57411, State: Running, Role: LEADER
I20260925 12:17:38.226089 7937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f84eb8c520a4b528af0e9fe52a57411" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34333 } }
I20260925 12:17:38.226397 7937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f84eb8c520a4b528af0e9fe52a57411" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f84eb8c520a4b528af0e9fe52a57411" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34333 } } }
I20260925 12:17:38.226485 7937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:38.226687 7938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f84eb8c520a4b528af0e9fe52a57411. Latest consensus state: current_term: 1 leader_uuid: "3f84eb8c520a4b528af0e9fe52a57411" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f84eb8c520a4b528af0e9fe52a57411" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34333 } } }
I20260925 12:17:38.226779 7938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:38.227361 7946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:38.227890 7946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:17:38.229329 7946 catalog_manager.cc:1347] Generated new cluster ID: 2765aa5ff1294674b75ed7998c695275
I20260925 12:17:38.229382 7946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:17:38 dist-test-slave-clmx krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:17:38.234560 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.225222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38798 (local address 127.0.98.62:34333)
0925 12:17:38.225392 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:38.225400 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:38.225430 (+ 30us) server_negotiation.cc:407] Connection header received
0925 12:17:38.225510 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:38.225515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:38.225550 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:38.225688 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:38.226428 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.228007 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.229016 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.229573 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.229687 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.231714 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:38.231718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:38.231718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:38.231728 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:38.233079 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:38.233471 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:38.233474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:38.233476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:38.233567 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:38.233777 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:38.233778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:38.233778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:38.234021 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:38.234179 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:38.234262 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 12:17:38.234389 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260925 12:17:38.239524 7946 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:17:38.240164 7946 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:38.245683 7946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411: Generated new TSK 0
I20260925 12:17:38.245849 7946 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260925 12:17:38.272065 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34333
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:38.364198 7958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:38.364403 7958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:38.364441 7958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:38.365558 7958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:38.365629 7958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:38.365660 7958 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 12:17:38.365694 7958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:17:38.366962 7958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:38.367193 7958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:38.367393 7958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:38.372380 7963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:38.372463 7966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:38.372375 7964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:38.372613 7958 server_base.cc:1034] running on GCE node
I20260925 12:17:38.372784 7958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:38.373041 7958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:38.374184 7958 hybrid_clock.cc:648] HybridClock initialized: now 1790338658374176 us; error 26 us; skew 500 ppm
Sep 25 12:17:38 dist-test-slave-clmx krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338658, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:38.376490 7958 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 12:17:38.376888 7958 webserver.cc:460] Webserver started at http://127.0.98.1:45495/ using document root <none> and password file <none>
I20260925 12:17:38.377056 7958 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:38.377092 7958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:38.377169 7958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:38.378043 7958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "02eeb6fd25e64aee8e2d44453426cc4d"
format_stamp: "Formatted at 2026-09-25 12:17:38 on dist-test-slave-clmx"
server_key: "562482ecf4fdf5aa3bf3949519d05805"
server_key_iv: "947ede7f1406aee59a1f6af2acba366f"
server_key_version: "encryptionkey@0"
I20260925 12:17:38.378293 7958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "02eeb6fd25e64aee8e2d44453426cc4d"
format_stamp: "Formatted at 2026-09-25 12:17:38 on dist-test-slave-clmx"
server_key: "562482ecf4fdf5aa3bf3949519d05805"
server_key_iv: "947ede7f1406aee59a1f6af2acba366f"
server_key_version: "encryptionkey@0"
I20260925 12:17:38.379349 7958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:17:38.379985 7973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:38.380119 7958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:38.380191 7958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "02eeb6fd25e64aee8e2d44453426cc4d"
format_stamp: "Formatted at 2026-09-25 12:17:38 on dist-test-slave-clmx"
server_key: "562482ecf4fdf5aa3bf3949519d05805"
server_key_iv: "947ede7f1406aee59a1f6af2acba366f"
server_key_version: "encryptionkey@0"
I20260925 12:17:38.380239 7958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:38.389719 7958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:38.391208 7958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:38.391348 7958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:38.391819 7958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:38.391876 7958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:38.391916 7958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:38.391939 7958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:38.397961 7958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46165
I20260925 12:17:38.398032 8088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46165 every 8 connection(s)
Sep 25 12:17:38 dist-test-slave-clmx krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338658, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:17:38.398310 7958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:17:38.401010 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.392137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41839 (local address 127.0.98.62:34333)
0925 12:17:38.392180 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:38.392182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:38.392354 (+ 172us) server_negotiation.cc:407] Connection header received
0925 12:17:38.392715 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:38.392717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:38.392730 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:38.392850 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:38.393513 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.394361 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.395806 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.396131 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.396212 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.398548 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:38.398549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:38.398550 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:38.398558 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:38.400007 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:38.400318 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:38.400319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:38.400319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:38.400366 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:38.400549 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:38.400550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:38.400550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:38.400666 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:38.400712 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:38.400900 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 12:17:38.400953 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:17:38.401188 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.392124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34333 (local address 127.0.98.1:41839)
0925 12:17:38.392246 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:38.392276 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:38.392634 (+ 358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:38.392912 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:38.392915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:38.393159 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:38.393410 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.393415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.394444 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.394444 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.395716 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.395718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.396244 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.396245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.396371 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.396875 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:38.396885 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:38.398455 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:38.400071 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:38.400074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:38.400075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:38.400257 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:38.400419 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:38.400419 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:38.400419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:38.400499 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:38.400715 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:38.400718 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:38.400810 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 12:17:38.400886 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 12:17:38.400984 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":64,"threads_started":1}
I20260925 12:17:38.401703 8089 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34333
I20260925 12:17:38.401798 8089 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:38.401949 8089 heartbeater.cc:511] Master 127.0.98.62:34333 requested a full tablet report, sending...
I20260925 12:17:38.402428 7886 ts_manager.cc:194] Registered new tserver with Master: 02eeb6fd25e64aee8e2d44453426cc4d (127.0.98.1:46165)
I20260925 12:17:38.403339 7886 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41839
I20260925 12:17:38.406567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 7958
I20260925 12:17:38.406646 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:17:38.406836 392 external_mini_cluster.cc:1428] Setting key 7c0ea8c6ded7df8011d9bebf33fa722f
I20260925 12:17:38.407699 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.402601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38421 (local address 127.0.98.62:34333)
0925 12:17:38.402638 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:38.402640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:38.402794 (+ 154us) server_negotiation.cc:407] Connection header received
0925 12:17:38.402834 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:38.402834 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:38.402841 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:38.402881 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:38.403185 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.403904 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.404555 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.404714 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.404766 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.405434 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:38.405435 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:38.405436 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:38.405440 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:38.406626 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:38.406978 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:38.406980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:38.406980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:38.407029 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:38.407220 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:38.407221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:38.407221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:38.407346 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:38.407400 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:38.407547 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 12:17:38.407604 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:17:38.407707 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.402563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34333 (local address 127.0.98.1:38421)
0925 12:17:38.402693 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:38.402708 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:38.402784 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:38.402896 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:38.402897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:38.403080 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:38.403129 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.403130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.404014 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.404015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.404486 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.404487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.404777 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.404778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.404818 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.405130 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:38.405134 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:38.405360 (+ 226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:38.406731 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:38.406732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:38.406732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:38.406910 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:38.407104 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:38.407105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:38.407105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:38.407165 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:38.407406 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:38.407407 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:38.407444 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 12:17:38.407513 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 12:17:38.407600 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260925 12:17:38.427732 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34333
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:38.501679 8098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:38.501863 8098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:38.501886 8098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:38.503046 8098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:38.503084 8098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:38.503105 8098 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 12:17:38.503122 8098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:17:38.504645 8098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:38.504818 8098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:38.504983 8098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:38.509423 8106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:38.509500 8098 server_base.cc:1034] running on GCE node
W20260925 12:17:38.509487 8104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:38.509433 8103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:38.509804 8098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:38.510018 8098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:38.511149 8098 hybrid_clock.cc:648] HybridClock initialized: now 1790338658511140 us; error 25 us; skew 500 ppm
Sep 25 12:17:38 dist-test-slave-clmx krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338658, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:38.518242 8098 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 12:17:38.518646 8098 webserver.cc:460] Webserver started at http://127.0.98.2:38599/ using document root <none> and password file <none>
I20260925 12:17:38.518841 8098 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:38.518910 8098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:38.519040 8098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:38.520045 8098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "7cff91078cef48fc9df4dd5b7dc48410"
format_stamp: "Formatted at 2026-09-25 12:17:38 on dist-test-slave-clmx"
server_key: "7c673b1a1f9ae46f3e4e7ffb7aada0a1"
server_key_iv: "0e0661d78a204f212d4b1ca03785e41d"
server_key_version: "encryptionkey@0"
I20260925 12:17:38.520376 8098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7cff91078cef48fc9df4dd5b7dc48410"
format_stamp: "Formatted at 2026-09-25 12:17:38 on dist-test-slave-clmx"
server_key: "7c673b1a1f9ae46f3e4e7ffb7aada0a1"
server_key_iv: "0e0661d78a204f212d4b1ca03785e41d"
server_key_version: "encryptionkey@0"
I20260925 12:17:38.521570 8098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:38.522236 8113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:38.522414 8098 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:38.522504 8098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "7cff91078cef48fc9df4dd5b7dc48410"
format_stamp: "Formatted at 2026-09-25 12:17:38 on dist-test-slave-clmx"
server_key: "7c673b1a1f9ae46f3e4e7ffb7aada0a1"
server_key_iv: "0e0661d78a204f212d4b1ca03785e41d"
server_key_version: "encryptionkey@0"
I20260925 12:17:38.522578 8098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:38.538481 8098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:38.540628 8098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:38.540812 8098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:38.541345 8098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:38.541402 8098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:38.541440 8098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:38.541463 8098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 12:17:38 dist-test-slave-clmx krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338658, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:17:38.548671 8098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45193
I20260925 12:17:38.548735 8228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45193 every 8 connection(s)
I20260925 12:17:38.549082 8098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:17:38.550405 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.541760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53469 (local address 127.0.98.62:34333)
0925 12:17:38.541823 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:38.541826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:38.541992 (+ 166us) server_negotiation.cc:407] Connection header received
0925 12:17:38.542429 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:38.542430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:38.542444 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:38.542522 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:38.543131 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.543661 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.544650 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.544939 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.545021 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.547395 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:38.547396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:38.547397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:38.547404 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:38.549147 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:38.549615 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:38.549617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:38.549617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:38.549678 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:38.549950 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:38.549950 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:38.549951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:38.550046 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:38.550094 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:38.550291 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 12:17:38.550347 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 12:17:38.550561 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.541735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34333 (local address 127.0.98.2:53469)
0925 12:17:38.541863 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:38.541901 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:38.542332 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:38.542578 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:38.542581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:38.542807 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:38.543037 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.543041 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.543744 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.543745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.544557 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.544560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.545031 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.545032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.545156 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.545640 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:38.545647 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:38.547268 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:38.549234 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:38.549238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:38.549239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:38.549543 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:38.549771 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:38.549772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:38.549772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:38.549890 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:38.550097 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:38.550101 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:38.550200 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 12:17:38.550274 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 12:17:38.550385 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":70,"threads_started":1}
I20260925 12:17:38.551096 8229 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34333
I20260925 12:17:38.551208 8229 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:38.551367 8229 heartbeater.cc:511] Master 127.0.98.62:34333 requested a full tablet report, sending...
I20260925 12:17:38.551419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 8098
I20260925 12:17:38.551519 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:17:38.551733 392 external_mini_cluster.cc:1428] Setting key 564d113035b0ce45146455d150878a8b
I20260925 12:17:38.551761 7885 ts_manager.cc:194] Registered new tserver with Master: 7cff91078cef48fc9df4dd5b7dc48410 (127.0.98.2:45193)
I20260925 12:17:38.552240 7885 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53469
I20260925 12:17:38.557950 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.552105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34333 (local address 127.0.98.2:55443)
0925 12:17:38.552215 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:38.552226 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:38.552278 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:38.552435 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:38.552436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:38.552647 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:38.552718 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.552720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.553441 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.553442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.554091 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.554093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.554514 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.554514 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.554553 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.554943 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:38.554947 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:38.555197 (+ 250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:38.556982 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:38.556983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:38.556983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:38.557133 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:38.557363 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:38.557364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:38.557364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:38.557411 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:38.557727 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:38.557728 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:38.557756 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 12:17:38.557800 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 12:17:38.557861 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":60,"threads_started":1}
I20260925 12:17:38.558022 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.552271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55443 (local address 127.0.98.62:34333)
0925 12:17:38.552304 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:38.552306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:38.552314 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:38.552332 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:38.552332 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:38.552341 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:38.552391 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:38.552823 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.553340 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.554165 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.554442 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.554511 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.555280 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:38.555282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:38.555282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:38.555289 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:38.556905 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:38.557210 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:38.557211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:38.557212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:38.557281 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:38.557468 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:38.557468 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:38.557469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:38.557664 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:38.557736 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:38.557825 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 12:17:38.557894 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 12:17:38.575328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34333
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:38.650506 8238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:38.650656 8238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:38.650708 8238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:38.651860 8238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:38.651896 8238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:38.651916 8238 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 12:17:38.651952 8238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:17:38.653307 8238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:38.653468 8238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:38.653616 8238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:38.657859 8246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:38.657884 8244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:38.657927 8243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:38.658048 8238 server_base.cc:1034] running on GCE node
I20260925 12:17:38.658219 8238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:38.658434 8238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:38.659584 8238 hybrid_clock.cc:648] HybridClock initialized: now 1790338658659572 us; error 22 us; skew 500 ppm
Sep 25 12:17:38 dist-test-slave-clmx krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338658, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:38.661811 8238 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 12:17:38.662189 8238 webserver.cc:460] Webserver started at http://127.0.98.3:46393/ using document root <none> and password file <none>
I20260925 12:17:38.662403 8238 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:38.662468 8238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:38.662608 8238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:38.663560 8238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559"
format_stamp: "Formatted at 2026-09-25 12:17:38 on dist-test-slave-clmx"
server_key: "57c8a8f655ef769d2e5b9afca20945d3"
server_key_iv: "a67e4b7c8b7c4728d9addad65aff3264"
server_key_version: "encryptionkey@0"
I20260925 12:17:38.663990 8238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559"
format_stamp: "Formatted at 2026-09-25 12:17:38 on dist-test-slave-clmx"
server_key: "57c8a8f655ef769d2e5b9afca20945d3"
server_key_iv: "a67e4b7c8b7c4728d9addad65aff3264"
server_key_version: "encryptionkey@0"
I20260925 12:17:38.665215 8238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:38.665867 8253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:38.666023 8238 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:38.666105 8238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559"
format_stamp: "Formatted at 2026-09-25 12:17:38 on dist-test-slave-clmx"
server_key: "57c8a8f655ef769d2e5b9afca20945d3"
server_key_iv: "a67e4b7c8b7c4728d9addad65aff3264"
server_key_version: "encryptionkey@0"
I20260925 12:17:38.666181 8238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:38.688113 8238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:38.689720 8238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:38.689883 8238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:38.690327 8238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:38.690388 8238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:38.690428 8238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:38.690455 8238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 12:17:38 dist-test-slave-clmx krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338658, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:17:38.697113 8238 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33569
I20260925 12:17:38.697175 8368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33569 every 8 connection(s)
I20260925 12:17:38.697450 8238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:17:38.699298 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 8238
I20260925 12:17:38.699409 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:17:38.699597 392 external_mini_cluster.cc:1428] Setting key 7de282dc7fc55cb70471b0d688236ff9
I20260925 12:17:38.699906 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.690861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54717 (local address 127.0.98.62:34333)
0925 12:17:38.690937 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:38.690940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:38.691074 (+ 134us) server_negotiation.cc:407] Connection header received
0925 12:17:38.691491 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:38.691493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:38.691504 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:38.691584 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:38.692260 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.693099 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.694639 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.694977 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.695054 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.697381 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:38.697382 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:38.697383 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:38.697389 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:38.698703 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:38.699022 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:38.699022 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:38.699023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:38.699064 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:38.699262 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:38.699263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:38.699264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:38.699374 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:38.699451 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:38.699735 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 12:17:38.699823 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 12:17:38.700069 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.690757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34333 (local address 127.0.98.3:54717)
0925 12:17:38.690951 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:38.690987 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:38.691398 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:38.691677 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:38.691681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:38.691909 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:38.692158 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.692164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.693197 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.693198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.694544 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.694546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.695088 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.695089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.695232 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.695702 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:38.695710 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:38.697292 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:38.698765 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:38.698768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:38.698769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:38.698963 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:38.699114 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:38.699115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:38.699115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:38.699193 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:38.699446 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:38.699452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:38.699590 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 12:17:38.699717 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 12:17:38.699875 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":108,"threads_started":1}
I20260925 12:17:38.700455 8369 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34333
I20260925 12:17:38.700527 8369 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:38.700667 8369 heartbeater.cc:511] Master 127.0.98.62:34333 requested a full tablet report, sending...
I20260925 12:17:38.700994 7885 ts_manager.cc:194] Registered new tserver with Master: dbd36d85cc7f4183a1c68ae2c6fd9559 (127.0.98.3:33569)
I20260925 12:17:38.701455 7885 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54717
I20260925 12:17:38.706627 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.701291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38811 (local address 127.0.98.62:34333)
0925 12:17:38.701322 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:38.701324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:38.701389 (+ 65us) server_negotiation.cc:407] Connection header received
0925 12:17:38.701445 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:38.701446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:38.701454 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:38.701508 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:38.701898 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.702383 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.703123 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.703320 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.703394 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.704278 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:38.704280 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:38.704280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:38.704286 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:38.705589 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:38.705916 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:38.705917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:38.705917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:38.705964 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:38.706196 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:38.706197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:38.706197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:38.706292 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:38.706351 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:38.706495 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 12:17:38.706555 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 12:17:38.706645 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.701236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34333 (local address 127.0.98.3:38811)
0925 12:17:38.701330 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:38.701338 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:38.701398 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:38.701528 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:38.701529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:38.701733 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:38.701786 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.701787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.702465 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.702466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.703034 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.703035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.703394 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.703395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.703461 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.703904 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:38.703908 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:38.704179 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:38.705687 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:38.705688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:38.705688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:38.705845 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:38.706035 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:38.706036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:38.706037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:38.706134 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:38.706355 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:38.706356 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:38.706399 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 12:17:38.706474 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 12:17:38.706556 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":55,"threads_started":1}
I20260925 12:17:38.711202 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 12:17:38 dist-test-slave-clmx krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338658, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 12:17:38 dist-test-slave-clmx krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:38 dist-test-slave-clmx krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 12:17:38.730341 7936 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:38.725305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.0.98.62:34333)
0925 12:17:38.725358 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:38.725359 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:38.725367 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:38.725418 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:38.725419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:38.725425 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:38.725470 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:38.725947 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.726505 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.727209 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.727441 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.727492 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.730260 (+ 2768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38814: BlockingRecv error: recv got EOF from 127.0.0.1:38814 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 12:17:38 dist-test-slave-clmx krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338658, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:17:38.736935 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.731178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38818 (local address 127.0.98.62:34333)
0925 12:17:38.731246 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:38.731248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:38.731328 (+ 80us) server_negotiation.cc:407] Connection header received
0925 12:17:38.731346 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:38.731346 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:38.731353 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:38.731431 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:38.731890 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.732289 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.732910 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.733070 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.733114 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.735040 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:38.735041 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:38.735042 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:38.735047 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:38.736156 (+ 1109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:38.736427 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:38.736428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:38.736428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:38.736468 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:38.736596 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:38.736596 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:38.736596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:38.736683 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:38.736742 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:38.736844 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 12:17:38.736888 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 12:17:38.738373 7885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:17:38.740470 7885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:17:38.746289 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.742408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33569 (local address 127.0.0.1:36540)
0925 12:17:38.742672 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:38.742681 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:38.742724 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:38.743413 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:38.743414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:38.743416 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:38.743457 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.743458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.744340 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.744341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.745291 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.745293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.746027 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.746028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.746091 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.746103 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 12:17:38.746174 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 12:17:38.746217 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":52,"thread_start_us":51,"threads_started":1}
I20260925 12:17:38.746308 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.742568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36540 (local address 127.0.98.3:33569)
0925 12:17:38.743213 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:38.743218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:38.743227 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:17:38.743258 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:38.743261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:38.743274 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:38.743354 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:38.743526 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.744252 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.745358 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.745949 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.746015 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.746075 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:38.746197 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 12:17:38.746230 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":68,"threads_started":1}
I20260925 12:17:38.746801 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.742367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46165 (local address 127.0.0.1:34342)
0925 12:17:38.742501 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:38.742536 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:38.742598 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:38.743090 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:38.743091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:38.743093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:38.743130 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.743131 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.744376 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.744377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.745602 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.745603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.746531 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.746532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.746611 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.746614 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:38.746689 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 12:17:38.746723 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":43,"thread_start_us":44,"threads_started":1}
I20260925 12:17:38.746858 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.742514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34342 (local address 127.0.98.1:46165)
0925 12:17:38.742988 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:38.742991 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:38.742998 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:17:38.743026 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:38.743028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:38.743038 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:38.743091 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:38.743214 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.744284 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.745679 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.746350 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.746412 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.746496 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:38.746704 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 12:17:38.746754 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":87,"threads_started":1}
I20260925 12:17:38.747097 8303 tablet_service.cc:1467] Processing CreateTablet for tablet b9f877c7dcd343d6b558d6c58855c91a (DEFAULT_TABLE table=test-table [id=540fc9048048483bbfff97d1e3a0db39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:38.747233 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.742585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57802 (local address 127.0.98.2:45193)
0925 12:17:38.742747 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:38.742751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:38.742796 (+ 45us) server_negotiation.cc:407] Connection header received
0925 12:17:38.742830 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:38.742832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:38.742844 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:38.742901 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:38.743029 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.744176 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.745690 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.746313 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.746526 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.746617 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:38.747102 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 12:17:38.747140 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260925 12:17:38.746894 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.742408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45193 (local address 127.0.0.1:57802)
0925 12:17:38.742695 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:38.742703 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:38.742747 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:38.742907 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:38.742909 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:38.742911 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:38.742951 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.742952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.744266 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.744267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.745600 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.745603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.746648 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.746649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.746716 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.746720 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:17:38.746786 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 12:17:38.746822 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":85,"thread_start_us":46,"threads_started":1}
I20260925 12:17:38.747457 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f877c7dcd343d6b558d6c58855c91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:38.747393 8023 tablet_service.cc:1467] Processing CreateTablet for tablet b9f877c7dcd343d6b558d6c58855c91a (DEFAULT_TABLE table=test-table [id=540fc9048048483bbfff97d1e3a0db39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:38.747746 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f877c7dcd343d6b558d6c58855c91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:38.747879 8163 tablet_service.cc:1467] Processing CreateTablet for tablet b9f877c7dcd343d6b558d6c58855c91a (DEFAULT_TABLE table=test-table [id=540fc9048048483bbfff97d1e3a0db39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:38.748162 8163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f877c7dcd343d6b558d6c58855c91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:38.750893 8394 tablet_bootstrap.cc:492] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d: Bootstrap starting.
I20260925 12:17:38.751444 8394 tablet_bootstrap.cc:654] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:38.751566 8396 tablet_bootstrap.cc:492] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410: Bootstrap starting.
I20260925 12:17:38.751796 8394 log.cc:824] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:38.751842 8395 tablet_bootstrap.cc:492] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559: Bootstrap starting.
I20260925 12:17:38.752149 8396 tablet_bootstrap.cc:654] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:38.752382 8395 tablet_bootstrap.cc:654] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:38.752444 8394 tablet_bootstrap.cc:492] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d: No bootstrap required, opened a new log
I20260925 12:17:38.752526 8394 ts_tablet_manager.cc:1397] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 12:17:38.752517 8396 log.cc:824] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:38.752692 8395 log.cc:824] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:38.753291 8396 tablet_bootstrap.cc:492] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410: No bootstrap required, opened a new log
I20260925 12:17:38.753340 8396 ts_tablet_manager.cc:1397] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 12:17:38.753520 8395 tablet_bootstrap.cc:492] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559: No bootstrap required, opened a new log
I20260925 12:17:38.753618 8395 ts_tablet_manager.cc:1397] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:17:38.754567 8394 raft_consensus.cc:348] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:38.754715 8394 raft_consensus.cc:374] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:38.754761 8394 raft_consensus.cc:729] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02eeb6fd25e64aee8e2d44453426cc4d, State: Initialized, Role: FOLLOWER
I20260925 12:17:38.754730 8396 raft_consensus.cc:348] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:38.754907 8396 raft_consensus.cc:374] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:38.754909 8394 consensus_queue.cc:260] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:38.754952 8396 raft_consensus.cc:729] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cff91078cef48fc9df4dd5b7dc48410, State: Initialized, Role: FOLLOWER
I20260925 12:17:38.755054 8396 consensus_queue.cc:260] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:38.755151 8394 ts_tablet_manager.cc:1428] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 12:17:38.755270 8089 heartbeater.cc:503] Master 127.0.98.62:34333 was elected leader, sending a full tablet report...
I20260925 12:17:38.755287 8396 ts_tablet_manager.cc:1428] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 12:17:38.755406 8229 heartbeater.cc:503] Master 127.0.98.62:34333 was elected leader, sending a full tablet report...
I20260925 12:17:38.755581 8395 raft_consensus.cc:348] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:38.755739 8395 raft_consensus.cc:374] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:38.755779 8395 raft_consensus.cc:729] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd36d85cc7f4183a1c68ae2c6fd9559, State: Initialized, Role: FOLLOWER
I20260925 12:17:38.755875 8395 consensus_queue.cc:260] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:38.756100 8395 ts_tablet_manager.cc:1428] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:17:38.756196 8369 heartbeater.cc:503] Master 127.0.98.62:34333 was elected leader, sending a full tablet report...
W20260925 12:17:38.799667 8230 tablet.cc:2369] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:17:38.875996 8401 raft_consensus.cc:482] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:17:38.876132 8401 raft_consensus.cc:504] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:38.876485 8401 leader_election.cc:288] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02eeb6fd25e64aee8e2d44453426cc4d (127.0.98.1:46165), dbd36d85cc7f4183a1c68ae2c6fd9559 (127.0.98.3:33569)
I20260925 12:17:38.879024 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.876695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50729 (local address 127.0.98.1:46165)
0925 12:17:38.876742 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:38.876745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:38.876776 (+ 31us) server_negotiation.cc:407] Connection header received
0925 12:17:38.876804 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:38.876806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:38.876817 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:38.876881 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:38.877034 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.877475 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.878336 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.878648 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.878726 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.878751 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:38.878914 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 12:17:38.878945 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 12:17:38.879045 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.876633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46165 (local address 127.0.98.2:50729)
0925 12:17:38.876672 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:38.876694 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:38.876740 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:38.876899 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:38.876900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:38.876902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:38.876966 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.876968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.877550 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.877551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.878247 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.878248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.878726 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.878727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.878827 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.878830 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:38.878908 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 12:17:38.878945 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 12:17:38.879103 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.876760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49809 (local address 127.0.98.3:33569)
0925 12:17:38.876796 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:38.876797 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:38.876894 (+ 97us) server_negotiation.cc:407] Connection header received
0925 12:17:38.876955 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:38.876956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:38.876965 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:38.877036 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:38.877174 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.877590 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.878419 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.878728 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.878832 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.878840 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:38.879001 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 12:17:38.879045 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 12:17:38.879128 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.876658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33569 (local address 127.0.98.2:49809)
0925 12:17:38.876825 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:38.876831 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:38.876902 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:38.877040 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:38.877041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:38.877042 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:38.877113 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.877114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.877699 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.877699 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.878338 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:38.878339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.878835 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.878836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:38.878919 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.878922 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:38.878996 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 12:17:38.879045 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":105,"threads_started":1}
I20260925 12:17:38.879374 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f877c7dcd343d6b558d6c58855c91a" candidate_uuid: "7cff91078cef48fc9df4dd5b7dc48410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" is_pre_election: true
I20260925 12:17:38.879395 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f877c7dcd343d6b558d6c58855c91a" candidate_uuid: "7cff91078cef48fc9df4dd5b7dc48410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" is_pre_election: true
I20260925 12:17:38.879508 8043 raft_consensus.cc:2454] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cff91078cef48fc9df4dd5b7dc48410 in term 0.
I20260925 12:17:38.879508 8323 raft_consensus.cc:2454] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cff91078cef48fc9df4dd5b7dc48410 in term 0.
I20260925 12:17:38.879732 8117 leader_election.cc:302] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02eeb6fd25e64aee8e2d44453426cc4d, 7cff91078cef48fc9df4dd5b7dc48410; no voters:
I20260925 12:17:38.879865 8401 raft_consensus.cc:2781] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:17:38.879918 8401 raft_consensus.cc:482] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:17:38.879941 8401 raft_consensus.cc:3037] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:38.880766 8401 raft_consensus.cc:504] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:38.880891 8401 leader_election.cc:288] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [CANDIDATE]: Term 1 election: Requested vote from peers 02eeb6fd25e64aee8e2d44453426cc4d (127.0.98.1:46165), dbd36d85cc7f4183a1c68ae2c6fd9559 (127.0.98.3:33569)
I20260925 12:17:38.881040 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f877c7dcd343d6b558d6c58855c91a" candidate_uuid: "7cff91078cef48fc9df4dd5b7dc48410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559"
I20260925 12:17:38.881110 8323 raft_consensus.cc:3037] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:38.881762 8323 raft_consensus.cc:2454] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cff91078cef48fc9df4dd5b7dc48410 in term 1.
I20260925 12:17:38.881873 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f877c7dcd343d6b558d6c58855c91a" candidate_uuid: "7cff91078cef48fc9df4dd5b7dc48410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02eeb6fd25e64aee8e2d44453426cc4d"
I20260925 12:17:38.881914 8115 leader_election.cc:302] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cff91078cef48fc9df4dd5b7dc48410, dbd36d85cc7f4183a1c68ae2c6fd9559; no voters:
I20260925 12:17:38.881951 8043 raft_consensus.cc:3037] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:38.882005 8401 raft_consensus.cc:2781] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:38.882139 8401 raft_consensus.cc:686] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [term 1 LEADER]: Becoming Leader. State: Replica: 7cff91078cef48fc9df4dd5b7dc48410, State: Running, Role: LEADER
I20260925 12:17:38.882241 8401 consensus_queue.cc:237] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:38.882809 8043 raft_consensus.cc:2454] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cff91078cef48fc9df4dd5b7dc48410 in term 1.
I20260925 12:17:38.883296 7885 catalog_manager.cc:5495] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cff91078cef48fc9df4dd5b7dc48410 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7cff91078cef48fc9df4dd5b7dc48410" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } health_report { overall_health: UNKNOWN } } }
I20260925 12:17:38.888981 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:38.886777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57816 (local address 127.0.98.2:45193)
0925 12:17:38.886826 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:38.886828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:38.886837 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:17:38.886878 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:38.886879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:38.886895 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:38.886960 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:38.887090 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.887505 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.888243 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:38.888437 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:38.888504 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:38.888676 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:38.888784 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:38.888877 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:38.888895 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 12:17:38.888912 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 12:17:38.890483 8043 raft_consensus.cc:1261] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d [term 1 FOLLOWER]: Refusing update from remote peer 7cff91078cef48fc9df4dd5b7dc48410: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:38.890735 8323 raft_consensus.cc:1261] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 1 FOLLOWER]: Refusing update from remote peer 7cff91078cef48fc9df4dd5b7dc48410: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:38.890800 8404 consensus_queue.cc:1035] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:38.890976 8404 consensus_queue.cc:1035] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:38.893020 8407 mvcc.cc:205] Tried to move back new op lower bound from 7333227146812973056 to 7333227146782187520. Current Snapshot: MvccSnapshot[applied={T|T < 7333227146812973056}]
I20260925 12:17:38.893282 8409 mvcc.cc:205] Tried to move back new op lower bound from 7333227146812973056 to 7333227146782187520. Current Snapshot: MvccSnapshot[applied={T|T < 7333227146812973056}]
I20260925 12:17:38.893877 8408 mvcc.cc:205] Tried to move back new op lower bound from 7333227146812973056 to 7333227146782187520. Current Snapshot: MvccSnapshot[applied={T|T < 7333227146812973056}]
I20260925 12:17:38.895092 7885 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:38.895188 7885 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:38.895889 7885 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411: Sending DeleteTablet for 3 replicas of tablet b9f877c7dcd343d6b558d6c58855c91a
I20260925 12:17:38.896199 8303 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f877c7dcd343d6b558d6c58855c91a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36540
I20260925 12:17:38.896219 8163 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f877c7dcd343d6b558d6c58855c91a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57802
I20260925 12:17:38.896214 8023 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f877c7dcd343d6b558d6c58855c91a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34342
I20260925 12:17:38.896420 8419 tablet_replica.cc:330] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d: stopping tablet replica
I20260925 12:17:38.896515 8419 raft_consensus.cc:2229] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:38.896610 8419 raft_consensus.cc:2258] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:38.896584 7885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:17:38.896711 7885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:17:38.896879 8418 tablet_replica.cc:330] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410: stopping tablet replica
I20260925 12:17:38.896965 8418 raft_consensus.cc:2229] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:17:38.897027 8419 ts_tablet_manager.cc:1905] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:38.897754 8418 raft_consensus.cc:2258] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:38.897959 8303 tablet_service.cc:1467] Processing CreateTablet for tablet 3842f33260ef40998f7303c6e7a85553 (DEFAULT_TABLE table=test-table [id=e5a196dcec1f4e389e3d448c155acbc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:38.898062 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3842f33260ef40998f7303c6e7a85553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:38.898170 8418 ts_tablet_manager.cc:1905] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:38.898542 8023 tablet_service.cc:1467] Processing CreateTablet for tablet 3842f33260ef40998f7303c6e7a85553 (DEFAULT_TABLE table=test-table [id=e5a196dcec1f4e389e3d448c155acbc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:38.898854 8163 tablet_service.cc:1467] Processing CreateTablet for tablet 3842f33260ef40998f7303c6e7a85553 (DEFAULT_TABLE table=test-table [id=e5a196dcec1f4e389e3d448c155acbc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:38.898959 8163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3842f33260ef40998f7303c6e7a85553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:38.899796 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3842f33260ef40998f7303c6e7a85553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:38.900120 8418 ts_tablet_manager.cc:1918] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:17:38.900149 8396 tablet_bootstrap.cc:492] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410: Bootstrap starting.
I20260925 12:17:38.900185 8418 log.cc:1196] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/b9f877c7dcd343d6b558d6c58855c91a
I20260925 12:17:38.900416 8418 ts_tablet_manager.cc:1939] T b9f877c7dcd343d6b558d6c58855c91a P 7cff91078cef48fc9df4dd5b7dc48410: Deleting consensus metadata
I20260925 12:17:38.900535 8396 tablet_bootstrap.cc:654] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:38.900952 7872 catalog_manager.cc:4841] TS 7cff91078cef48fc9df4dd5b7dc48410 (127.0.98.2:45193): tablet b9f877c7dcd343d6b558d6c58855c91a (table test-table [id=540fc9048048483bbfff97d1e3a0db39]) successfully deleted
I20260925 12:17:38.900971 8417 tablet_replica.cc:330] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559: stopping tablet replica
I20260925 12:17:38.901054 8417 raft_consensus.cc:2229] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:38.901062 8394 tablet_bootstrap.cc:492] T 3842f33260ef40998f7303c6e7a85553 P 02eeb6fd25e64aee8e2d44453426cc4d: Bootstrap starting.
I20260925 12:17:38.901160 8417 raft_consensus.cc:2258] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:38.901475 8394 tablet_bootstrap.cc:654] T 3842f33260ef40998f7303c6e7a85553 P 02eeb6fd25e64aee8e2d44453426cc4d: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:38.901690 8417 ts_tablet_manager.cc:1905] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:38.902154 8396 tablet_bootstrap.cc:492] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410: No bootstrap required, opened a new log
I20260925 12:17:38.902191 8396 ts_tablet_manager.cc:1397] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 12:17:38.902344 8396 raft_consensus.cc:348] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:38.902398 8396 raft_consensus.cc:374] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:38.902419 8396 raft_consensus.cc:729] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cff91078cef48fc9df4dd5b7dc48410, State: Initialized, Role: FOLLOWER
I20260925 12:17:38.902460 8396 consensus_queue.cc:260] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:38.902467 8395 tablet_bootstrap.cc:492] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559: Bootstrap starting.
I20260925 12:17:38.902531 8396 ts_tablet_manager.cc:1428] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:38.902869 8395 tablet_bootstrap.cc:654] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:38.903244 8417 ts_tablet_manager.cc:1918] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:17:38.903317 8417 log.cc:1196] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/b9f877c7dcd343d6b558d6c58855c91a
I20260925 12:17:38.903561 8417 ts_tablet_manager.cc:1939] T b9f877c7dcd343d6b558d6c58855c91a P dbd36d85cc7f4183a1c68ae2c6fd9559: Deleting consensus metadata
I20260925 12:17:38.903672 8395 tablet_bootstrap.cc:492] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559: No bootstrap required, opened a new log
I20260925 12:17:38.903710 8395 ts_tablet_manager.cc:1397] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:38.903831 8395 raft_consensus.cc:348] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:38.903888 8395 raft_consensus.cc:374] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:38.903908 8395 raft_consensus.cc:729] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd36d85cc7f4183a1c68ae2c6fd9559, State: Initialized, Role: FOLLOWER
I20260925 12:17:38.903949 8395 consensus_queue.cc:260] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:38.904034 8395 ts_tablet_manager.cc:1428] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:38.904181 7871 catalog_manager.cc:4841] TS dbd36d85cc7f4183a1c68ae2c6fd9559 (127.0.98.3:33569): tablet b9f877c7dcd343d6b558d6c58855c91a (table test-table [id=540fc9048048483bbfff97d1e3a0db39]) successfully deleted
I20260925 12:17:38.904343 8419 ts_tablet_manager.cc:1918] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:17:38.904410 8419 log.cc:1196] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-0/wal/wals/b9f877c7dcd343d6b558d6c58855c91a
I20260925 12:17:38.904641 8419 ts_tablet_manager.cc:1939] T b9f877c7dcd343d6b558d6c58855c91a P 02eeb6fd25e64aee8e2d44453426cc4d: Deleting consensus metadata
I20260925 12:17:38.905022 7873 catalog_manager.cc:4841] TS 02eeb6fd25e64aee8e2d44453426cc4d (127.0.98.1:46165): tablet b9f877c7dcd343d6b558d6c58855c91a (table test-table [id=540fc9048048483bbfff97d1e3a0db39]) successfully deleted
I20260925 12:17:38.907955 8394 tablet_bootstrap.cc:492] T 3842f33260ef40998f7303c6e7a85553 P 02eeb6fd25e64aee8e2d44453426cc4d: No bootstrap required, opened a new log
I20260925 12:17:38.908015 8394 ts_tablet_manager.cc:1397] T 3842f33260ef40998f7303c6e7a85553 P 02eeb6fd25e64aee8e2d44453426cc4d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260925 12:17:38.908133 8394 raft_consensus.cc:348] T 3842f33260ef40998f7303c6e7a85553 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:38.908196 8394 raft_consensus.cc:374] T 3842f33260ef40998f7303c6e7a85553 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:38.908226 8394 raft_consensus.cc:729] T 3842f33260ef40998f7303c6e7a85553 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02eeb6fd25e64aee8e2d44453426cc4d, State: Initialized, Role: FOLLOWER
I20260925 12:17:38.908273 8394 consensus_queue.cc:260] T 3842f33260ef40998f7303c6e7a85553 P 02eeb6fd25e64aee8e2d44453426cc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:38.908350 8394 ts_tablet_manager.cc:1428] T 3842f33260ef40998f7303c6e7a85553 P 02eeb6fd25e64aee8e2d44453426cc4d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260925 12:17:38.947929 8370 tablet.cc:2369] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:17:39.009156 8404 raft_consensus.cc:482] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:17:39.009259 8404 raft_consensus.cc:504] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.009534 8404 leader_election.cc:288] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02eeb6fd25e64aee8e2d44453426cc4d (127.0.98.1:46165), dbd36d85cc7f4183a1c68ae2c6fd9559 (127.0.98.3:33569)
I20260925 12:17:39.009761 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3842f33260ef40998f7303c6e7a85553" candidate_uuid: "7cff91078cef48fc9df4dd5b7dc48410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" is_pre_election: true
I20260925 12:17:39.009761 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3842f33260ef40998f7303c6e7a85553" candidate_uuid: "7cff91078cef48fc9df4dd5b7dc48410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" is_pre_election: true
I20260925 12:17:39.009855 8323 raft_consensus.cc:2454] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cff91078cef48fc9df4dd5b7dc48410 in term 0.
I20260925 12:17:39.009855 8043 raft_consensus.cc:2454] T 3842f33260ef40998f7303c6e7a85553 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cff91078cef48fc9df4dd5b7dc48410 in term 0.
I20260925 12:17:39.009997 8115 leader_election.cc:302] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cff91078cef48fc9df4dd5b7dc48410, dbd36d85cc7f4183a1c68ae2c6fd9559; no voters:
I20260925 12:17:39.010095 8404 raft_consensus.cc:2781] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:17:39.010138 8404 raft_consensus.cc:482] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:17:39.010169 8404 raft_consensus.cc:3037] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:39.010816 8404 raft_consensus.cc:504] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.010949 8404 leader_election.cc:288] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [CANDIDATE]: Term 1 election: Requested vote from peers 02eeb6fd25e64aee8e2d44453426cc4d (127.0.98.1:46165), dbd36d85cc7f4183a1c68ae2c6fd9559 (127.0.98.3:33569)
I20260925 12:17:39.011101 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3842f33260ef40998f7303c6e7a85553" candidate_uuid: "7cff91078cef48fc9df4dd5b7dc48410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02eeb6fd25e64aee8e2d44453426cc4d"
I20260925 12:17:39.011152 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3842f33260ef40998f7303c6e7a85553" candidate_uuid: "7cff91078cef48fc9df4dd5b7dc48410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559"
I20260925 12:17:39.011242 8323 raft_consensus.cc:3037] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:39.011255 8043 raft_consensus.cc:3037] T 3842f33260ef40998f7303c6e7a85553 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:39.011893 8323 raft_consensus.cc:2454] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cff91078cef48fc9df4dd5b7dc48410 in term 1.
I20260925 12:17:39.011906 8043 raft_consensus.cc:2454] T 3842f33260ef40998f7303c6e7a85553 P 02eeb6fd25e64aee8e2d44453426cc4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cff91078cef48fc9df4dd5b7dc48410 in term 1.
I20260925 12:17:39.012071 8117 leader_election.cc:302] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02eeb6fd25e64aee8e2d44453426cc4d, 7cff91078cef48fc9df4dd5b7dc48410; no voters:
I20260925 12:17:39.012200 8404 raft_consensus.cc:2781] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:39.012254 8404 raft_consensus.cc:686] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [term 1 LEADER]: Becoming Leader. State: Replica: 7cff91078cef48fc9df4dd5b7dc48410, State: Running, Role: LEADER
I20260925 12:17:39.012306 8404 consensus_queue.cc:237] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.012825 7885 catalog_manager.cc:5495] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cff91078cef48fc9df4dd5b7dc48410 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7cff91078cef48fc9df4dd5b7dc48410" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } health_report { overall_health: UNKNOWN } } }
Sep 25 12:17:39 dist-test-slave-clmx krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338658, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:17:39.042510 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:39.036147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34333 (local address 127.0.0.1:38830)
0925 12:17:39.036305 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:39.036315 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:39.036396 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:39.036550 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:39.036550 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:39.036842 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:39.036879 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:39.036881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.037532 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.037533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:39.038041 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:39.038042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.038399 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.038400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:39.038454 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:39.038995 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:39.038999 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:39.040236 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:39.041495 (+ 1259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:39.041501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:39.041503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:39.041672 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:39.041880 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:39.041881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:39.041881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:39.041932 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:39.042161 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:39.042167 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:39.042209 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 12:17:39.042285 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 12:17:39.042397 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260925 12:17:39.042518 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:39.036215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38830 (local address 127.0.98.62:34333)
0925 12:17:39.036296 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:39.036298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:39.036387 (+ 89us) server_negotiation.cc:407] Connection header received
0925 12:17:39.036461 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:39.036462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:39.036470 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:39.036533 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:39.036977 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.037455 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.038125 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.038320 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.038398 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:39.040322 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:39.040323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:39.040323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:39.040330 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:39.041416 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:39.041752 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:39.041753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:39.041754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:39.041811 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:39.041988 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:39.041989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:39.041989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:39.042082 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:39.042170 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:39.042325 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 12:17:39.042443 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":51}
I20260925 12:17:39.044351 7885 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38830:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 12:17:39.044530 7885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:17:39.045745 8023 tablet_service.cc:1467] Processing CreateTablet for tablet 0976d47f56234e5eb08ddabd81984ae9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f917e633bd5f44d78438376f494e9bc4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 12:17:39.045737 8163 tablet_service.cc:1467] Processing CreateTablet for tablet 0976d47f56234e5eb08ddabd81984ae9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f917e633bd5f44d78438376f494e9bc4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 12:17:39.045774 8303 tablet_service.cc:1467] Processing CreateTablet for tablet 0976d47f56234e5eb08ddabd81984ae9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f917e633bd5f44d78438376f494e9bc4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 12:17:39.045871 8163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0976d47f56234e5eb08ddabd81984ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:39.045871 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0976d47f56234e5eb08ddabd81984ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:39.045910 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0976d47f56234e5eb08ddabd81984ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:39.046931 8395 tablet_bootstrap.cc:492] T 0976d47f56234e5eb08ddabd81984ae9 P dbd36d85cc7f4183a1c68ae2c6fd9559: Bootstrap starting.
I20260925 12:17:39.047202 8394 tablet_bootstrap.cc:492] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d: Bootstrap starting.
I20260925 12:17:39.047324 8395 tablet_bootstrap.cc:654] T 0976d47f56234e5eb08ddabd81984ae9 P dbd36d85cc7f4183a1c68ae2c6fd9559: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:39.047595 8394 tablet_bootstrap.cc:654] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:39.047942 8396 tablet_bootstrap.cc:492] T 0976d47f56234e5eb08ddabd81984ae9 P 7cff91078cef48fc9df4dd5b7dc48410: Bootstrap starting.
I20260925 12:17:39.048074 8395 tablet_bootstrap.cc:492] T 0976d47f56234e5eb08ddabd81984ae9 P dbd36d85cc7f4183a1c68ae2c6fd9559: No bootstrap required, opened a new log
I20260925 12:17:39.048110 8395 ts_tablet_manager.cc:1397] T 0976d47f56234e5eb08ddabd81984ae9 P dbd36d85cc7f4183a1c68ae2c6fd9559: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 12:17:39.048255 8396 tablet_bootstrap.cc:654] T 0976d47f56234e5eb08ddabd81984ae9 P 7cff91078cef48fc9df4dd5b7dc48410: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:39.048251 8395 raft_consensus.cc:348] T 0976d47f56234e5eb08ddabd81984ae9 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.048307 8395 raft_consensus.cc:374] T 0976d47f56234e5eb08ddabd81984ae9 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:39.048343 8395 raft_consensus.cc:729] T 0976d47f56234e5eb08ddabd81984ae9 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd36d85cc7f4183a1c68ae2c6fd9559, State: Initialized, Role: FOLLOWER
I20260925 12:17:39.048412 8395 consensus_queue.cc:260] T 0976d47f56234e5eb08ddabd81984ae9 P dbd36d85cc7f4183a1c68ae2c6fd9559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.048523 8395 ts_tablet_manager.cc:1428] T 0976d47f56234e5eb08ddabd81984ae9 P dbd36d85cc7f4183a1c68ae2c6fd9559: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:39.048590 8402 tablet_replica.cc:439] T 0976d47f56234e5eb08ddabd81984ae9 P dbd36d85cc7f4183a1c68ae2c6fd9559: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } } }
I20260925 12:17:39.048682 8402 tablet_replica.cc:442] T 0976d47f56234e5eb08ddabd81984ae9 P dbd36d85cc7f4183a1c68ae2c6fd9559: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:17:39.048801 8396 tablet_bootstrap.cc:492] T 0976d47f56234e5eb08ddabd81984ae9 P 7cff91078cef48fc9df4dd5b7dc48410: No bootstrap required, opened a new log
I20260925 12:17:39.048832 8394 tablet_bootstrap.cc:492] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d: No bootstrap required, opened a new log
I20260925 12:17:39.048838 8396 ts_tablet_manager.cc:1397] T 0976d47f56234e5eb08ddabd81984ae9 P 7cff91078cef48fc9df4dd5b7dc48410: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 12:17:39.048866 8394 ts_tablet_manager.cc:1397] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:17:39.048946 8396 raft_consensus.cc:348] T 0976d47f56234e5eb08ddabd81984ae9 P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.048996 8396 raft_consensus.cc:374] T 0976d47f56234e5eb08ddabd81984ae9 P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:39.048975 8394 raft_consensus.cc:348] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.049013 8396 raft_consensus.cc:729] T 0976d47f56234e5eb08ddabd81984ae9 P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cff91078cef48fc9df4dd5b7dc48410, State: Initialized, Role: FOLLOWER
I20260925 12:17:39.049028 8394 raft_consensus.cc:374] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:39.049065 8394 raft_consensus.cc:729] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02eeb6fd25e64aee8e2d44453426cc4d, State: Initialized, Role: FOLLOWER
I20260925 12:17:39.049070 8396 consensus_queue.cc:260] T 0976d47f56234e5eb08ddabd81984ae9 P 7cff91078cef48fc9df4dd5b7dc48410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.049116 8394 consensus_queue.cc:260] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.049158 8396 ts_tablet_manager.cc:1428] T 0976d47f56234e5eb08ddabd81984ae9 P 7cff91078cef48fc9df4dd5b7dc48410: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:39.049188 8394 ts_tablet_manager.cc:1428] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:39.049225 8404 tablet_replica.cc:439] T 0976d47f56234e5eb08ddabd81984ae9 P 7cff91078cef48fc9df4dd5b7dc48410: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } } }
I20260925 12:17:39.049271 8404 tablet_replica.cc:442] T 0976d47f56234e5eb08ddabd81984ae9 P 7cff91078cef48fc9df4dd5b7dc48410: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:17:39.049233 8400 tablet_replica.cc:439] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } } }
I20260925 12:17:39.049332 8400 tablet_replica.cc:442] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 12:17:39.149097 8090 tablet.cc:2369] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:17:39.165663 8400 raft_consensus.cc:482] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:17:39.165735 8400 raft_consensus.cc:504] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.166083 8400 leader_election.cc:288] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cff91078cef48fc9df4dd5b7dc48410 (127.0.98.2:45193), dbd36d85cc7f4183a1c68ae2c6fd9559 (127.0.98.3:33569)
I20260925 12:17:39.168813 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:39.166342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40971 (local address 127.0.98.2:45193)
0925 12:17:39.166391 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:39.166393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:39.166443 (+ 50us) server_negotiation.cc:407] Connection header received
0925 12:17:39.166503 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:39.166504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:39.166514 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:39.166597 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:39.166787 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.167201 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.168117 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.168486 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.168561 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:39.168571 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:39.168722 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 12:17:39.168754 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 12:17:39.168895 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:39.166275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45193 (local address 127.0.98.1:40971)
0925 12:17:39.166381 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:39.166392 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:39.166445 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:39.166599 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:39.166599 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:39.166601 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:39.166698 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:39.166699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.167296 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.167297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:39.168032 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:39.168034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.168560 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.168561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:39.168632 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:39.168635 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:39.168718 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 12:17:39.168788 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":18,"thread_start_us":46,"threads_started":1}
I20260925 12:17:39.168900 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:39.166389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37219 (local address 127.0.98.3:33569)
0925 12:17:39.166439 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:39.166441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:39.166450 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:17:39.166501 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:39.166502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:39.166511 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:39.166594 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:17:39.166746 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.167176 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.168198 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.168549 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.168635 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:39.168643 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:39.168789 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 12:17:39.168820 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 12:17:39.168995 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:39.166263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33569 (local address 127.0.98.1:37219)
0925 12:17:39.166372 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:39.166385 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:39.166445 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:39.166596 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:39.166597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:39.166599 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:17:39.166679 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:39.166681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.167248 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.167248 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:39.168101 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:39.168103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.168638 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.168639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:39.168716 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:39.168719 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:39.168783 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 12:17:39.168818 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":17,"thread_start_us":58,"threads_started":1}
I20260925 12:17:39.169227 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0976d47f56234e5eb08ddabd81984ae9" candidate_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" is_pre_election: true
I20260925 12:17:39.169317 8323 raft_consensus.cc:2454] T 0976d47f56234e5eb08ddabd81984ae9 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02eeb6fd25e64aee8e2d44453426cc4d in term 0.
I20260925 12:17:39.169494 7975 leader_election.cc:302] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02eeb6fd25e64aee8e2d44453426cc4d, dbd36d85cc7f4183a1c68ae2c6fd9559; no voters:
I20260925 12:17:39.169508 8183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0976d47f56234e5eb08ddabd81984ae9" candidate_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cff91078cef48fc9df4dd5b7dc48410" is_pre_election: true
I20260925 12:17:39.169610 8400 raft_consensus.cc:2781] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:17:39.169646 8400 raft_consensus.cc:482] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:17:39.169646 8183 raft_consensus.cc:2454] T 0976d47f56234e5eb08ddabd81984ae9 P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02eeb6fd25e64aee8e2d44453426cc4d in term 0.
I20260925 12:17:39.169672 8400 raft_consensus.cc:3037] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:39.170342 8400 raft_consensus.cc:504] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.170468 8400 leader_election.cc:288] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d [CANDIDATE]: Term 1 election: Requested vote from peers 7cff91078cef48fc9df4dd5b7dc48410 (127.0.98.2:45193), dbd36d85cc7f4183a1c68ae2c6fd9559 (127.0.98.3:33569)
I20260925 12:17:39.170616 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0976d47f56234e5eb08ddabd81984ae9" candidate_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559"
I20260925 12:17:39.170632 8183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0976d47f56234e5eb08ddabd81984ae9" candidate_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cff91078cef48fc9df4dd5b7dc48410"
I20260925 12:17:39.170717 8183 raft_consensus.cc:3037] T 0976d47f56234e5eb08ddabd81984ae9 P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:39.170717 8323 raft_consensus.cc:3037] T 0976d47f56234e5eb08ddabd81984ae9 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:39.171240 8323 raft_consensus.cc:2454] T 0976d47f56234e5eb08ddabd81984ae9 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02eeb6fd25e64aee8e2d44453426cc4d in term 1.
I20260925 12:17:39.171365 8183 raft_consensus.cc:2454] T 0976d47f56234e5eb08ddabd81984ae9 P 7cff91078cef48fc9df4dd5b7dc48410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02eeb6fd25e64aee8e2d44453426cc4d in term 1.
I20260925 12:17:39.171452 7975 leader_election.cc:302] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02eeb6fd25e64aee8e2d44453426cc4d, dbd36d85cc7f4183a1c68ae2c6fd9559; no voters:
I20260925 12:17:39.171548 8400 raft_consensus.cc:2781] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:39.171671 8400 raft_consensus.cc:686] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d [term 1 LEADER]: Becoming Leader. State: Replica: 02eeb6fd25e64aee8e2d44453426cc4d, State: Running, Role: LEADER
I20260925 12:17:39.171734 8400 consensus_queue.cc:237] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.172055 8435 tablet_replica.cc:439] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d: TxnStatusTablet state changed. Reason: New leader 02eeb6fd25e64aee8e2d44453426cc4d. Latest consensus state: current_term: 1 leader_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } } }
I20260925 12:17:39.172143 8435 tablet_replica.cc:442] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d: This TxnStatusTablet replica's current role is: LEADER
I20260925 12:17:39.172310 8437 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 12:17:39.172421 8437 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 12:17:39.172411 7885 catalog_manager.cc:5495] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 02eeb6fd25e64aee8e2d44453426cc4d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } health_report { overall_health: UNKNOWN } } }
I20260925 12:17:39.195756 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:39.193430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.0.98.1:46165)
0925 12:17:39.193473 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:39.193475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:39.193487 (+ 12us) server_negotiation.cc:407] Connection header received
0925 12:17:39.193580 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:39.193581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:39.193597 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:39.193675 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:39.193814 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.194276 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.194981 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.195171 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.195240 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:39.195424 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:39.195511 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:39.195584 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:39.195675 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 12:17:39.195693 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:17:39.195808 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:39.193381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46165 (local address 127.0.0.1:34348)
0925 12:17:39.193447 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:39.193460 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:39.193506 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:39.193681 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:39.193682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:39.193682 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:17:39.193737 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:39.193739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.194359 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.194360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:39.194900 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:39.194902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.195252 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.195254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:39.195313 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:39.195338 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:17:39.195581 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:17:39.195589 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 12:17:39.195669 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 12:17:39.195713 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 12:17:39.196837 8323 raft_consensus.cc:1261] T 0976d47f56234e5eb08ddabd81984ae9 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 1 FOLLOWER]: Refusing update from remote peer 02eeb6fd25e64aee8e2d44453426cc4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:39.196906 8183 raft_consensus.cc:1261] T 0976d47f56234e5eb08ddabd81984ae9 P 7cff91078cef48fc9df4dd5b7dc48410 [term 1 FOLLOWER]: Refusing update from remote peer 02eeb6fd25e64aee8e2d44453426cc4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:39.197068 8435 consensus_queue.cc:1035] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:39.197173 8400 consensus_queue.cc:1035] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:39.197678 8402 tablet_replica.cc:439] T 0976d47f56234e5eb08ddabd81984ae9 P dbd36d85cc7f4183a1c68ae2c6fd9559: TxnStatusTablet state changed. Reason: New leader 02eeb6fd25e64aee8e2d44453426cc4d. Latest consensus state: current_term: 1 leader_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } } }
I20260925 12:17:39.197791 8402 tablet_replica.cc:442] T 0976d47f56234e5eb08ddabd81984ae9 P dbd36d85cc7f4183a1c68ae2c6fd9559: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:17:39.198284 8404 tablet_replica.cc:439] T 0976d47f56234e5eb08ddabd81984ae9 P 7cff91078cef48fc9df4dd5b7dc48410: TxnStatusTablet state changed. Reason: New leader 02eeb6fd25e64aee8e2d44453426cc4d. Latest consensus state: current_term: 1 leader_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } } }
I20260925 12:17:39.198362 8404 tablet_replica.cc:442] T 0976d47f56234e5eb08ddabd81984ae9 P 7cff91078cef48fc9df4dd5b7dc48410: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:17:39.198325 8400 tablet_replica.cc:439] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } } }
I20260925 12:17:39.198397 8400 tablet_replica.cc:442] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d: This TxnStatusTablet replica's current role is: LEADER
I20260925 12:17:39.198668 8400 tablet_replica.cc:439] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } } }
I20260925 12:17:39.198724 8400 tablet_replica.cc:442] T 0976d47f56234e5eb08ddabd81984ae9 P 02eeb6fd25e64aee8e2d44453426cc4d: This TxnStatusTablet replica's current role is: LEADER
I20260925 12:17:39.198868 8402 tablet_replica.cc:439] T 0976d47f56234e5eb08ddabd81984ae9 P dbd36d85cc7f4183a1c68ae2c6fd9559: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } } }
I20260925 12:17:39.198936 8402 tablet_replica.cc:442] T 0976d47f56234e5eb08ddabd81984ae9 P dbd36d85cc7f4183a1c68ae2c6fd9559: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:17:39.199726 8404 tablet_replica.cc:439] T 0976d47f56234e5eb08ddabd81984ae9 P 7cff91078cef48fc9df4dd5b7dc48410: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } } }
I20260925 12:17:39.199798 8404 tablet_replica.cc:442] T 0976d47f56234e5eb08ddabd81984ae9 P 7cff91078cef48fc9df4dd5b7dc48410: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:17:39.204048 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:39.201538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52079 (local address 127.0.98.1:46165)
0925 12:17:39.201569 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:39.201570 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:39.201701 (+ 131us) server_negotiation.cc:407] Connection header received
0925 12:17:39.201760 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:39.201760 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:39.201770 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:39.201822 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:39.201934 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.202444 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.203356 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.203557 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.203654 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:39.203809 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:39.203838 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:39.203906 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:39.203967 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 12:17:39.203994 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 12:17:39.204097 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:39.201459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46165 (local address 127.0.98.2:52079)
0925 12:17:39.201636 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:39.201645 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:39.201715 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:39.201831 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:39.201832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:39.201834 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:17:39.201873 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:39.201874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.202527 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.202528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:39.203255 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:39.203257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.203645 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.203646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:39.203700 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:39.203713 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:17:39.203908 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:17:39.203911 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:39.203963 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 12:17:39.204018 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":79,"threads_started":1}
I20260925 12:17:39.205974 8043 raft_consensus.cc:1261] T 3842f33260ef40998f7303c6e7a85553 P 02eeb6fd25e64aee8e2d44453426cc4d [term 1 FOLLOWER]: Refusing update from remote peer 7cff91078cef48fc9df4dd5b7dc48410: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:39.206111 8323 raft_consensus.cc:1261] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 1 FOLLOWER]: Refusing update from remote peer 7cff91078cef48fc9df4dd5b7dc48410: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:39.206162 8404 consensus_queue.cc:1035] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:39.206261 8412 consensus_queue.cc:1035] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:39.214174 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:39.211258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52029 (local address 127.0.98.2:45193)
0925 12:17:39.211287 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:39.211288 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:39.211395 (+ 107us) server_negotiation.cc:407] Connection header received
0925 12:17:39.211554 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:39.211555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:39.211566 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:39.211593 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:39.211838 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.212251 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.213432 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.213702 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.213746 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:39.213966 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:39.213996 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:39.214051 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:39.214104 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 12:17:39.214120 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 12:17:39.214210 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:39.211226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45193 (local address 127.0.98.1:52029)
0925 12:17:39.211347 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:39.211356 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:39.211432 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:39.211718 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:39.211719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:39.211721 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:17:39.211780 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:39.211781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.212330 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.212331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:39.213321 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:39.213325 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.213797 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.213799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:39.213879 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:39.213892 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:17:39.214053 (+ 161us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:17:39.214054 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 12:17:39.214100 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 12:17:39.214131 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260925 12:17:39.219518 7884 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:39.219640 7884 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:39.220836 7884 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411: Sending DeleteTablet for 3 replicas of tablet 3842f33260ef40998f7303c6e7a85553
I20260925 12:17:39.221101 8163 tablet_service.cc:1514] Processing DeleteTablet for tablet 3842f33260ef40998f7303c6e7a85553 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57802
I20260925 12:17:39.221115 8303 tablet_service.cc:1514] Processing DeleteTablet for tablet 3842f33260ef40998f7303c6e7a85553 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36540
I20260925 12:17:39.221187 8418 tablet_replica.cc:330] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410: stopping tablet replica
I20260925 12:17:39.221228 8418 raft_consensus.cc:2229] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:17:39.221273 8418 raft_consensus.cc:2258] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:39.221333 8417 tablet_replica.cc:330] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559: stopping tablet replica
I20260925 12:17:39.221386 8417 raft_consensus.cc:2229] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:39.221417 8417 raft_consensus.cc:2258] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:39.221547 8023 tablet_service.cc:1514] Processing DeleteTablet for tablet 3842f33260ef40998f7303c6e7a85553 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34342
I20260925 12:17:39.221612 8419 tablet_replica.cc:330] T 3842f33260ef40998f7303c6e7a85553 P 02eeb6fd25e64aee8e2d44453426cc4d: stopping tablet replica
I20260925 12:17:39.221644 8419 raft_consensus.cc:2229] T 3842f33260ef40998f7303c6e7a85553 P 02eeb6fd25e64aee8e2d44453426cc4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:39.221673 8419 raft_consensus.cc:2258] T 3842f33260ef40998f7303c6e7a85553 P 02eeb6fd25e64aee8e2d44453426cc4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:39.221964 8419 ts_tablet_manager.cc:1905] T 3842f33260ef40998f7303c6e7a85553 P 02eeb6fd25e64aee8e2d44453426cc4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:39.222019 8418 ts_tablet_manager.cc:1905] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:39.223184 8417 ts_tablet_manager.cc:1905] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:39.223601 8418 ts_tablet_manager.cc:1918] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:17:39.223667 8419 ts_tablet_manager.cc:1918] T 3842f33260ef40998f7303c6e7a85553 P 02eeb6fd25e64aee8e2d44453426cc4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:17:39.223687 8418 log.cc:1196] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/3842f33260ef40998f7303c6e7a85553
I20260925 12:17:39.223713 8419 log.cc:1196] T 3842f33260ef40998f7303c6e7a85553 P 02eeb6fd25e64aee8e2d44453426cc4d: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-0/wal/wals/3842f33260ef40998f7303c6e7a85553
I20260925 12:17:39.223922 8418 ts_tablet_manager.cc:1939] T 3842f33260ef40998f7303c6e7a85553 P 7cff91078cef48fc9df4dd5b7dc48410: Deleting consensus metadata
I20260925 12:17:39.223928 8419 ts_tablet_manager.cc:1939] T 3842f33260ef40998f7303c6e7a85553 P 02eeb6fd25e64aee8e2d44453426cc4d: Deleting consensus metadata
I20260925 12:17:39.224246 7873 catalog_manager.cc:4841] TS 02eeb6fd25e64aee8e2d44453426cc4d (127.0.98.1:46165): tablet 3842f33260ef40998f7303c6e7a85553 (table test-table [id=e5a196dcec1f4e389e3d448c155acbc9]) successfully deleted
I20260925 12:17:39.224318 7872 catalog_manager.cc:4841] TS 7cff91078cef48fc9df4dd5b7dc48410 (127.0.98.2:45193): tablet 3842f33260ef40998f7303c6e7a85553 (table test-table [id=e5a196dcec1f4e389e3d448c155acbc9]) successfully deleted
I20260925 12:17:39.224648 8417 ts_tablet_manager.cc:1918] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:17:39.224694 8417 log.cc:1196] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/3842f33260ef40998f7303c6e7a85553
I20260925 12:17:39.224932 8417 ts_tablet_manager.cc:1939] T 3842f33260ef40998f7303c6e7a85553 P dbd36d85cc7f4183a1c68ae2c6fd9559: Deleting consensus metadata
I20260925 12:17:39.225188 7871 catalog_manager.cc:4841] TS dbd36d85cc7f4183a1c68ae2c6fd9559 (127.0.98.3:33569): tablet 3842f33260ef40998f7303c6e7a85553 (table test-table [id=e5a196dcec1f4e389e3d448c155acbc9]) successfully deleted
Sep 25 12:17:39 dist-test-slave-clmx krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 12:17:39 dist-test-slave-clmx krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:39 dist-test-slave-clmx krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 12:17:39.246894 7936 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:39.241122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38842 (local address 127.0.98.62:34333)
0925 12:17:39.241186 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:39.241188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:39.241338 (+ 150us) server_negotiation.cc:407] Connection header received
0925 12:17:39.241401 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:39.241402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:39.241410 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:39.241456 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:39.242161 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.242602 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.243693 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.243868 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.243917 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:39.246816 (+ 2899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38842: BlockingRecv error: recv got EOF from 127.0.0.1:38842 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 12:17:39 dist-test-slave-clmx krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:17:39.254879 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:39.247690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38856 (local address 127.0.98.62:34333)
0925 12:17:39.247728 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:39.247730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:39.247737 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:17:39.247792 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:39.247792 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:39.247798 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:39.247839 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:39.248321 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.248700 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.249572 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.250006 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.250079 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:39.252446 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:39.252447 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:39.252448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:39.252454 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:39.253792 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:39.254176 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:39.254177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:39.254178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:39.254229 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:39.254424 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:39.254425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:39.254425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:39.254525 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:39.254621 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:39.254718 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 12:17:39.254782 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 12:17:39.256354 7884 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:17:39.256482 7884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:17:39.257685 8163 tablet_service.cc:1467] Processing CreateTablet for tablet 21019de50f2c4f42baada7125396ffc8 (DEFAULT_TABLE table=test-table [id=3ffa96f81fb140d9a1963d300aba2e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:39.257697 8303 tablet_service.cc:1467] Processing CreateTablet for tablet 21019de50f2c4f42baada7125396ffc8 (DEFAULT_TABLE table=test-table [id=3ffa96f81fb140d9a1963d300aba2e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:39.257694 8023 tablet_service.cc:1467] Processing CreateTablet for tablet 21019de50f2c4f42baada7125396ffc8 (DEFAULT_TABLE table=test-table [id=3ffa96f81fb140d9a1963d300aba2e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:39.257795 8163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21019de50f2c4f42baada7125396ffc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:39.257795 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21019de50f2c4f42baada7125396ffc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:39.257800 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21019de50f2c4f42baada7125396ffc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:39.258935 8396 tablet_bootstrap.cc:492] T 21019de50f2c4f42baada7125396ffc8 P 7cff91078cef48fc9df4dd5b7dc48410: Bootstrap starting.
I20260925 12:17:39.259032 8395 tablet_bootstrap.cc:492] T 21019de50f2c4f42baada7125396ffc8 P dbd36d85cc7f4183a1c68ae2c6fd9559: Bootstrap starting.
I20260925 12:17:39.259209 8396 tablet_bootstrap.cc:654] T 21019de50f2c4f42baada7125396ffc8 P 7cff91078cef48fc9df4dd5b7dc48410: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:39.259387 8395 tablet_bootstrap.cc:654] T 21019de50f2c4f42baada7125396ffc8 P dbd36d85cc7f4183a1c68ae2c6fd9559: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:39.259682 8394 tablet_bootstrap.cc:492] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d: Bootstrap starting.
I20260925 12:17:39.259720 8396 tablet_bootstrap.cc:492] T 21019de50f2c4f42baada7125396ffc8 P 7cff91078cef48fc9df4dd5b7dc48410: No bootstrap required, opened a new log
I20260925 12:17:39.259763 8396 ts_tablet_manager.cc:1397] T 21019de50f2c4f42baada7125396ffc8 P 7cff91078cef48fc9df4dd5b7dc48410: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 12:17:39.259871 8396 raft_consensus.cc:348] T 21019de50f2c4f42baada7125396ffc8 P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.259912 8396 raft_consensus.cc:374] T 21019de50f2c4f42baada7125396ffc8 P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:39.259923 8396 raft_consensus.cc:729] T 21019de50f2c4f42baada7125396ffc8 P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cff91078cef48fc9df4dd5b7dc48410, State: Initialized, Role: FOLLOWER
I20260925 12:17:39.259976 8396 consensus_queue.cc:260] T 21019de50f2c4f42baada7125396ffc8 P 7cff91078cef48fc9df4dd5b7dc48410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.260015 8394 tablet_bootstrap.cc:654] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:39.260071 8396 ts_tablet_manager.cc:1428] T 21019de50f2c4f42baada7125396ffc8 P 7cff91078cef48fc9df4dd5b7dc48410: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:39.260624 8394 tablet_bootstrap.cc:492] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d: No bootstrap required, opened a new log
I20260925 12:17:39.260666 8394 ts_tablet_manager.cc:1397] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:39.260808 8394 raft_consensus.cc:348] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.260885 8394 raft_consensus.cc:374] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:39.260917 8394 raft_consensus.cc:729] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02eeb6fd25e64aee8e2d44453426cc4d, State: Initialized, Role: FOLLOWER
I20260925 12:17:39.260963 8394 consensus_queue.cc:260] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.261049 8394 ts_tablet_manager.cc:1428] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:39.261119 8395 tablet_bootstrap.cc:492] T 21019de50f2c4f42baada7125396ffc8 P dbd36d85cc7f4183a1c68ae2c6fd9559: No bootstrap required, opened a new log
I20260925 12:17:39.261174 8395 ts_tablet_manager.cc:1397] T 21019de50f2c4f42baada7125396ffc8 P dbd36d85cc7f4183a1c68ae2c6fd9559: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 12:17:39.261318 8395 raft_consensus.cc:348] T 21019de50f2c4f42baada7125396ffc8 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.261408 8395 raft_consensus.cc:374] T 21019de50f2c4f42baada7125396ffc8 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:39.261430 8395 raft_consensus.cc:729] T 21019de50f2c4f42baada7125396ffc8 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd36d85cc7f4183a1c68ae2c6fd9559, State: Initialized, Role: FOLLOWER
I20260925 12:17:39.261466 8395 consensus_queue.cc:260] T 21019de50f2c4f42baada7125396ffc8 P dbd36d85cc7f4183a1c68ae2c6fd9559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.261548 8395 ts_tablet_manager.cc:1428] T 21019de50f2c4f42baada7125396ffc8 P dbd36d85cc7f4183a1c68ae2c6fd9559: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:39.723695 8400 raft_consensus.cc:482] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:17:39.723796 8400 raft_consensus.cc:504] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.723963 8400 leader_election.cc:288] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cff91078cef48fc9df4dd5b7dc48410 (127.0.98.2:45193), dbd36d85cc7f4183a1c68ae2c6fd9559 (127.0.98.3:33569)
I20260925 12:17:39.724169 8183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21019de50f2c4f42baada7125396ffc8" candidate_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cff91078cef48fc9df4dd5b7dc48410" is_pre_election: true
I20260925 12:17:39.724190 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21019de50f2c4f42baada7125396ffc8" candidate_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" is_pre_election: true
I20260925 12:17:39.724274 8183 raft_consensus.cc:2454] T 21019de50f2c4f42baada7125396ffc8 P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02eeb6fd25e64aee8e2d44453426cc4d in term 0.
I20260925 12:17:39.724284 8323 raft_consensus.cc:2454] T 21019de50f2c4f42baada7125396ffc8 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02eeb6fd25e64aee8e2d44453426cc4d in term 0.
I20260925 12:17:39.724424 7976 leader_election.cc:302] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02eeb6fd25e64aee8e2d44453426cc4d, 7cff91078cef48fc9df4dd5b7dc48410; no voters:
I20260925 12:17:39.724519 8400 raft_consensus.cc:2781] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:17:39.724573 8400 raft_consensus.cc:482] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:17:39.724594 8400 raft_consensus.cc:3037] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:39.725351 8400 raft_consensus.cc:504] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.725479 8400 leader_election.cc:288] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [CANDIDATE]: Term 1 election: Requested vote from peers 7cff91078cef48fc9df4dd5b7dc48410 (127.0.98.2:45193), dbd36d85cc7f4183a1c68ae2c6fd9559 (127.0.98.3:33569)
I20260925 12:17:39.725589 8183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21019de50f2c4f42baada7125396ffc8" candidate_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cff91078cef48fc9df4dd5b7dc48410"
I20260925 12:17:39.725603 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21019de50f2c4f42baada7125396ffc8" candidate_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559"
I20260925 12:17:39.725662 8183 raft_consensus.cc:3037] T 21019de50f2c4f42baada7125396ffc8 P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:39.725662 8323 raft_consensus.cc:3037] T 21019de50f2c4f42baada7125396ffc8 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:39.726408 8323 raft_consensus.cc:2454] T 21019de50f2c4f42baada7125396ffc8 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02eeb6fd25e64aee8e2d44453426cc4d in term 1.
I20260925 12:17:39.726403 8183 raft_consensus.cc:2454] T 21019de50f2c4f42baada7125396ffc8 P 7cff91078cef48fc9df4dd5b7dc48410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02eeb6fd25e64aee8e2d44453426cc4d in term 1.
I20260925 12:17:39.726559 7976 leader_election.cc:302] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02eeb6fd25e64aee8e2d44453426cc4d, 7cff91078cef48fc9df4dd5b7dc48410; no voters:
I20260925 12:17:39.726641 8400 raft_consensus.cc:2781] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:39.726744 8400 raft_consensus.cc:686] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [term 1 LEADER]: Becoming Leader. State: Replica: 02eeb6fd25e64aee8e2d44453426cc4d, State: Running, Role: LEADER
I20260925 12:17:39.726796 8400 consensus_queue.cc:237] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.727334 7886 catalog_manager.cc:5495] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 02eeb6fd25e64aee8e2d44453426cc4d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } health_report { overall_health: UNKNOWN } } }
I20260925 12:17:39.809129 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:39.807214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34364 (local address 127.0.98.1:46165)
0925 12:17:39.807334 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:39.807336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:39.807343 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:17:39.807362 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:39.807363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:39.807372 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:39.807425 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:39.807523 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.807961 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.808622 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.808804 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.808852 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:39.808954 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:39.808975 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:39.809007 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:39.809059 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 12:17:39.809072 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":72,"threads_started":1}
I20260925 12:17:39.809978 8183 raft_consensus.cc:1261] T 21019de50f2c4f42baada7125396ffc8 P 7cff91078cef48fc9df4dd5b7dc48410 [term 1 FOLLOWER]: Refusing update from remote peer 02eeb6fd25e64aee8e2d44453426cc4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:39.810137 8323 raft_consensus.cc:1261] T 21019de50f2c4f42baada7125396ffc8 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 1 FOLLOWER]: Refusing update from remote peer 02eeb6fd25e64aee8e2d44453426cc4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:39.810230 8466 consensus_queue.cc:1035] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:39.810369 8400 consensus_queue.cc:1035] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:39.813684 7886 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:39.813768 7886 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:39.814389 7886 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411: Sending DeleteTablet for 3 replicas of tablet 21019de50f2c4f42baada7125396ffc8
I20260925 12:17:39.814646 8163 tablet_service.cc:1514] Processing DeleteTablet for tablet 21019de50f2c4f42baada7125396ffc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57802
I20260925 12:17:39.814713 8023 tablet_service.cc:1514] Processing DeleteTablet for tablet 21019de50f2c4f42baada7125396ffc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34342
I20260925 12:17:39.814880 8484 tablet_replica.cc:330] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d: stopping tablet replica
I20260925 12:17:39.814877 8303 tablet_service.cc:1514] Processing DeleteTablet for tablet 21019de50f2c4f42baada7125396ffc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36540
I20260925 12:17:39.814858 7886 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:17:39.815004 7886 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:17:39.814951 8484 raft_consensus.cc:2229] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:17:39.815119 8485 tablet_replica.cc:330] T 21019de50f2c4f42baada7125396ffc8 P dbd36d85cc7f4183a1c68ae2c6fd9559: stopping tablet replica
I20260925 12:17:39.815142 8484 raft_consensus.cc:2258] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:39.815191 8485 raft_consensus.cc:2229] T 21019de50f2c4f42baada7125396ffc8 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:39.815259 8485 raft_consensus.cc:2258] T 21019de50f2c4f42baada7125396ffc8 P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:39.815490 8485 ts_tablet_manager.cc:1905] T 21019de50f2c4f42baada7125396ffc8 P dbd36d85cc7f4183a1c68ae2c6fd9559: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:39.816202 8484 ts_tablet_manager.cc:1905] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:39.816220 8163 tablet_service.cc:1467] Processing CreateTablet for tablet 328968991cf4452ca2d02d46e8179e1d (DEFAULT_TABLE table=test-table [id=355c691d6b33458bb1797bbcc9cb73bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:39.816287 8023 tablet_service.cc:1467] Processing CreateTablet for tablet 328968991cf4452ca2d02d46e8179e1d (DEFAULT_TABLE table=test-table [id=355c691d6b33458bb1797bbcc9cb73bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:39.816321 8163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 328968991cf4452ca2d02d46e8179e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:39.816388 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 328968991cf4452ca2d02d46e8179e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:39.817764 8303 tablet_service.cc:1467] Processing CreateTablet for tablet 328968991cf4452ca2d02d46e8179e1d (DEFAULT_TABLE table=test-table [id=355c691d6b33458bb1797bbcc9cb73bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:17:39.817873 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 328968991cf4452ca2d02d46e8179e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:39.818156 8484 ts_tablet_manager.cc:1918] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:17:39.818229 8484 log.cc:1196] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-0/wal/wals/21019de50f2c4f42baada7125396ffc8
I20260925 12:17:39.818441 8484 ts_tablet_manager.cc:1939] T 21019de50f2c4f42baada7125396ffc8 P 02eeb6fd25e64aee8e2d44453426cc4d: Deleting consensus metadata
I20260925 12:17:39.819092 8488 tablet_bootstrap.cc:492] T 328968991cf4452ca2d02d46e8179e1d P dbd36d85cc7f4183a1c68ae2c6fd9559: Bootstrap starting.
I20260925 12:17:39.819284 7873 catalog_manager.cc:4841] TS 02eeb6fd25e64aee8e2d44453426cc4d (127.0.98.1:46165): tablet 21019de50f2c4f42baada7125396ffc8 (table test-table [id=3ffa96f81fb140d9a1963d300aba2e7b]) successfully deleted
I20260925 12:17:39.819691 8488 tablet_bootstrap.cc:654] T 328968991cf4452ca2d02d46e8179e1d P dbd36d85cc7f4183a1c68ae2c6fd9559: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:39.819711 8483 tablet_replica.cc:330] T 21019de50f2c4f42baada7125396ffc8 P 7cff91078cef48fc9df4dd5b7dc48410: stopping tablet replica
I20260925 12:17:39.819871 8483 raft_consensus.cc:2229] T 21019de50f2c4f42baada7125396ffc8 P 7cff91078cef48fc9df4dd5b7dc48410 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:39.820046 8483 raft_consensus.cc:2258] T 21019de50f2c4f42baada7125396ffc8 P 7cff91078cef48fc9df4dd5b7dc48410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:39.820282 8483 ts_tablet_manager.cc:1905] T 21019de50f2c4f42baada7125396ffc8 P 7cff91078cef48fc9df4dd5b7dc48410: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:39.820276 8485 ts_tablet_manager.cc:1918] T 21019de50f2c4f42baada7125396ffc8 P dbd36d85cc7f4183a1c68ae2c6fd9559: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:17:39.820351 8485 log.cc:1196] T 21019de50f2c4f42baada7125396ffc8 P dbd36d85cc7f4183a1c68ae2c6fd9559: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/21019de50f2c4f42baada7125396ffc8
I20260925 12:17:39.820565 8485 ts_tablet_manager.cc:1939] T 21019de50f2c4f42baada7125396ffc8 P dbd36d85cc7f4183a1c68ae2c6fd9559: Deleting consensus metadata
I20260925 12:17:39.820890 7871 catalog_manager.cc:4841] TS dbd36d85cc7f4183a1c68ae2c6fd9559 (127.0.98.3:33569): tablet 21019de50f2c4f42baada7125396ffc8 (table test-table [id=3ffa96f81fb140d9a1963d300aba2e7b]) successfully deleted
I20260925 12:17:39.821494 8483 ts_tablet_manager.cc:1918] T 21019de50f2c4f42baada7125396ffc8 P 7cff91078cef48fc9df4dd5b7dc48410: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:17:39.821578 8483 log.cc:1196] T 21019de50f2c4f42baada7125396ffc8 P 7cff91078cef48fc9df4dd5b7dc48410: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/21019de50f2c4f42baada7125396ffc8
I20260925 12:17:39.821818 8483 ts_tablet_manager.cc:1939] T 21019de50f2c4f42baada7125396ffc8 P 7cff91078cef48fc9df4dd5b7dc48410: Deleting consensus metadata
I20260925 12:17:39.822079 7872 catalog_manager.cc:4841] TS 7cff91078cef48fc9df4dd5b7dc48410 (127.0.98.2:45193): tablet 21019de50f2c4f42baada7125396ffc8 (table test-table [id=3ffa96f81fb140d9a1963d300aba2e7b]) successfully deleted
I20260925 12:17:39.823714 8486 tablet_bootstrap.cc:492] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d: Bootstrap starting.
I20260925 12:17:39.824041 8486 tablet_bootstrap.cc:654] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:39.824312 8487 tablet_bootstrap.cc:492] T 328968991cf4452ca2d02d46e8179e1d P 7cff91078cef48fc9df4dd5b7dc48410: Bootstrap starting.
I20260925 12:17:39.824627 8486 tablet_bootstrap.cc:492] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d: No bootstrap required, opened a new log
I20260925 12:17:39.824682 8486 ts_tablet_manager.cc:1397] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:39.824787 8486 raft_consensus.cc:348] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.824868 8486 raft_consensus.cc:374] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:39.824896 8486 raft_consensus.cc:729] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02eeb6fd25e64aee8e2d44453426cc4d, State: Initialized, Role: FOLLOWER
I20260925 12:17:39.824913 8487 tablet_bootstrap.cc:654] T 328968991cf4452ca2d02d46e8179e1d P 7cff91078cef48fc9df4dd5b7dc48410: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:39.824944 8486 consensus_queue.cc:260] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.825100 8486 ts_tablet_manager.cc:1428] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:39.825534 8487 tablet_bootstrap.cc:492] T 328968991cf4452ca2d02d46e8179e1d P 7cff91078cef48fc9df4dd5b7dc48410: No bootstrap required, opened a new log
I20260925 12:17:39.825577 8487 ts_tablet_manager.cc:1397] T 328968991cf4452ca2d02d46e8179e1d P 7cff91078cef48fc9df4dd5b7dc48410: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:39.825691 8487 raft_consensus.cc:348] T 328968991cf4452ca2d02d46e8179e1d P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.825753 8487 raft_consensus.cc:374] T 328968991cf4452ca2d02d46e8179e1d P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:39.825796 8487 raft_consensus.cc:729] T 328968991cf4452ca2d02d46e8179e1d P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cff91078cef48fc9df4dd5b7dc48410, State: Initialized, Role: FOLLOWER
I20260925 12:17:39.825847 8487 consensus_queue.cc:260] T 328968991cf4452ca2d02d46e8179e1d P 7cff91078cef48fc9df4dd5b7dc48410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.825997 8487 ts_tablet_manager.cc:1428] T 328968991cf4452ca2d02d46e8179e1d P 7cff91078cef48fc9df4dd5b7dc48410: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:39.826413 8488 tablet_bootstrap.cc:492] T 328968991cf4452ca2d02d46e8179e1d P dbd36d85cc7f4183a1c68ae2c6fd9559: No bootstrap required, opened a new log
I20260925 12:17:39.826467 8488 ts_tablet_manager.cc:1397] T 328968991cf4452ca2d02d46e8179e1d P dbd36d85cc7f4183a1c68ae2c6fd9559: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260925 12:17:39.826604 8488 raft_consensus.cc:348] T 328968991cf4452ca2d02d46e8179e1d P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.826675 8488 raft_consensus.cc:374] T 328968991cf4452ca2d02d46e8179e1d P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:39.826697 8488 raft_consensus.cc:729] T 328968991cf4452ca2d02d46e8179e1d P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd36d85cc7f4183a1c68ae2c6fd9559, State: Initialized, Role: FOLLOWER
I20260925 12:17:39.826779 8488 consensus_queue.cc:260] T 328968991cf4452ca2d02d46e8179e1d P dbd36d85cc7f4183a1c68ae2c6fd9559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.826869 8488 ts_tablet_manager.cc:1428] T 328968991cf4452ca2d02d46e8179e1d P dbd36d85cc7f4183a1c68ae2c6fd9559: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:39.853184 8466 raft_consensus.cc:482] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:17:39.853263 8466 raft_consensus.cc:504] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.853421 8466 leader_election.cc:288] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cff91078cef48fc9df4dd5b7dc48410 (127.0.98.2:45193), dbd36d85cc7f4183a1c68ae2c6fd9559 (127.0.98.3:33569)
I20260925 12:17:39.853539 8183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328968991cf4452ca2d02d46e8179e1d" candidate_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cff91078cef48fc9df4dd5b7dc48410" is_pre_election: true
I20260925 12:17:39.853550 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328968991cf4452ca2d02d46e8179e1d" candidate_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" is_pre_election: true
I20260925 12:17:39.853618 8183 raft_consensus.cc:2454] T 328968991cf4452ca2d02d46e8179e1d P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02eeb6fd25e64aee8e2d44453426cc4d in term 0.
I20260925 12:17:39.853627 8323 raft_consensus.cc:2454] T 328968991cf4452ca2d02d46e8179e1d P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02eeb6fd25e64aee8e2d44453426cc4d in term 0.
I20260925 12:17:39.853742 7975 leader_election.cc:302] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02eeb6fd25e64aee8e2d44453426cc4d, dbd36d85cc7f4183a1c68ae2c6fd9559; no voters:
I20260925 12:17:39.853869 8466 raft_consensus.cc:2781] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:17:39.853911 8466 raft_consensus.cc:482] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:17:39.853935 8466 raft_consensus.cc:3037] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:39.854485 8466 raft_consensus.cc:504] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.854624 8466 leader_election.cc:288] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d [CANDIDATE]: Term 1 election: Requested vote from peers 7cff91078cef48fc9df4dd5b7dc48410 (127.0.98.2:45193), dbd36d85cc7f4183a1c68ae2c6fd9559 (127.0.98.3:33569)
I20260925 12:17:39.854737 8183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328968991cf4452ca2d02d46e8179e1d" candidate_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cff91078cef48fc9df4dd5b7dc48410"
I20260925 12:17:39.854784 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328968991cf4452ca2d02d46e8179e1d" candidate_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559"
I20260925 12:17:39.854816 8183 raft_consensus.cc:3037] T 328968991cf4452ca2d02d46e8179e1d P 7cff91078cef48fc9df4dd5b7dc48410 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:39.854857 8323 raft_consensus.cc:3037] T 328968991cf4452ca2d02d46e8179e1d P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:39.855449 8323 raft_consensus.cc:2454] T 328968991cf4452ca2d02d46e8179e1d P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02eeb6fd25e64aee8e2d44453426cc4d in term 1.
I20260925 12:17:39.855449 8183 raft_consensus.cc:2454] T 328968991cf4452ca2d02d46e8179e1d P 7cff91078cef48fc9df4dd5b7dc48410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02eeb6fd25e64aee8e2d44453426cc4d in term 1.
I20260925 12:17:39.855581 7975 leader_election.cc:302] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02eeb6fd25e64aee8e2d44453426cc4d, dbd36d85cc7f4183a1c68ae2c6fd9559; no voters:
I20260925 12:17:39.855674 8466 raft_consensus.cc:2781] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:39.855712 8466 raft_consensus.cc:686] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d [term 1 LEADER]: Becoming Leader. State: Replica: 02eeb6fd25e64aee8e2d44453426cc4d, State: Running, Role: LEADER
I20260925 12:17:39.855749 8466 consensus_queue.cc:237] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } }
I20260925 12:17:39.856206 7886 catalog_manager.cc:5495] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 02eeb6fd25e64aee8e2d44453426cc4d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02eeb6fd25e64aee8e2d44453426cc4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 } health_report { overall_health: UNKNOWN } } }
I20260925 12:17:39.868033 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:39.865651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57611 (local address 127.0.98.1:46165)
0925 12:17:39.865685 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:39.865688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:39.865770 (+ 82us) server_negotiation.cc:407] Connection header received
0925 12:17:39.865836 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:39.865836 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:39.865844 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:39.865883 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:17:39.866002 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.866439 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.867433 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.867595 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.867709 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:39.867847 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:17:39.867871 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:17:39.867905 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:39.867955 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 12:17:39.867970 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 12:17:39.868106 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:39.865598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46165 (local address 127.0.98.1:57611)
0925 12:17:39.865717 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:39.865726 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:39.865778 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:39.865896 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:39.865897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:39.865899 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:17:39.865949 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:39.865951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.866507 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.866508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:39.867358 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:39.867360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:39.867711 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:39.867712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:39.867773 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:39.867778 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:17:39.867905 (+ 127us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:17:39.867906 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 12:17:39.867954 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 12:17:39.867992 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
I20260925 12:17:39.869516 8183 raft_consensus.cc:1261] T 328968991cf4452ca2d02d46e8179e1d P 7cff91078cef48fc9df4dd5b7dc48410 [term 1 FOLLOWER]: Refusing update from remote peer 02eeb6fd25e64aee8e2d44453426cc4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:39.869688 8478 consensus_queue.cc:1035] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cff91078cef48fc9df4dd5b7dc48410" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:39.869850 8323 raft_consensus.cc:1261] T 328968991cf4452ca2d02d46e8179e1d P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 1 FOLLOWER]: Refusing update from remote peer 02eeb6fd25e64aee8e2d44453426cc4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:17:39.870539 8478 consensus_queue.cc:1035] T 328968991cf4452ca2d02d46e8179e1d P 02eeb6fd25e64aee8e2d44453426cc4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbd36d85cc7f4183a1c68ae2c6fd9559" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:17:39.879180 7884 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:39.879274 7884 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:17:39.879858 7884 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f84eb8c520a4b528af0e9fe52a57411: Sending DeleteTablet for 3 replicas of tablet 328968991cf4452ca2d02d46e8179e1d
I20260925 12:17:39.880054 8163 tablet_service.cc:1514] Processing DeleteTablet for tablet 328968991cf4452ca2d02d46e8179e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57802
I20260925 12:17:39.880144 8483 tablet_replica.cc:330] T 328968991cf4452ca2d02d46e8179e1d P 7cff91078cef48fc9df4dd5b7dc48410: stopping tablet replica
I20260925 12:17:39.880131 8303 tablet_service.cc:1514] Processing DeleteTablet for tablet 328968991cf4452ca2d02d46e8179e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36540
I20260925 12:17:39.880185 8483 raft_consensus.cc:2229] T 328968991cf4452ca2d02d46e8179e1d P 7cff91078cef48fc9df4dd5b7dc48410 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:39.880216 8483 raft_consensus.cc:2258] T 328968991cf4452ca2d02d46e8179e1d P 7cff91078cef48fc9df4dd5b7dc48410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:39.880285 8485 tablet_replica.cc:330] T 328968991cf4452ca2d02d46e8179e1d P dbd36d85cc7f4183a1c68ae2c6fd9559: stopping tablet replica
I20260925 12:17:39.880340 8485 raft_consensus.cc:2229] T 328968991cf4452ca2d02d46e8179e1d P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:17:39.880380 8485 raft_consensus.cc:2258] T 328968991cf4452ca2d02d46e8179e1d P dbd36d85cc7f4183a1c68ae2c6fd9559 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:17:39.880445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 7958
I20260925 12:17:39.880501 8023 tablet_service.cc:1514] Processing DeleteTablet for tablet 328968991cf4452ca2d02d46e8179e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:17:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34342
I20260925 12:17:39.880726 8483 ts_tablet_manager.cc:1905] T 328968991cf4452ca2d02d46e8179e1d P 7cff91078cef48fc9df4dd5b7dc48410: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:39.882047 8483 ts_tablet_manager.cc:1918] T 328968991cf4452ca2d02d46e8179e1d P 7cff91078cef48fc9df4dd5b7dc48410: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:17:39.882104 8483 log.cc:1196] T 328968991cf4452ca2d02d46e8179e1d P 7cff91078cef48fc9df4dd5b7dc48410: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/328968991cf4452ca2d02d46e8179e1d
I20260925 12:17:39.882303 8483 ts_tablet_manager.cc:1939] T 328968991cf4452ca2d02d46e8179e1d P 7cff91078cef48fc9df4dd5b7dc48410: Deleting consensus metadata
I20260925 12:17:39.882642 8485 ts_tablet_manager.cc:1905] T 328968991cf4452ca2d02d46e8179e1d P dbd36d85cc7f4183a1c68ae2c6fd9559: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:17:39.882658 7872 catalog_manager.cc:4841] TS 7cff91078cef48fc9df4dd5b7dc48410 (127.0.98.2:45193): tablet 328968991cf4452ca2d02d46e8179e1d (table test-table [id=355c691d6b33458bb1797bbcc9cb73bc]) successfully deleted
I20260925 12:17:39.883678 8485 ts_tablet_manager.cc:1918] T 328968991cf4452ca2d02d46e8179e1d P dbd36d85cc7f4183a1c68ae2c6fd9559: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:17:39.883755 8485 log.cc:1196] T 328968991cf4452ca2d02d46e8179e1d P dbd36d85cc7f4183a1c68ae2c6fd9559: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/328968991cf4452ca2d02d46e8179e1d
I20260925 12:17:39.883948 8485 ts_tablet_manager.cc:1939] T 328968991cf4452ca2d02d46e8179e1d P dbd36d85cc7f4183a1c68ae2c6fd9559: Deleting consensus metadata
I20260925 12:17:39.884244 7871 catalog_manager.cc:4841] TS dbd36d85cc7f4183a1c68ae2c6fd9559 (127.0.98.3:33569): tablet 328968991cf4452ca2d02d46e8179e1d (table test-table [id=355c691d6b33458bb1797bbcc9cb73bc]) successfully deleted
W20260925 12:17:39.888430 7873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46165 (error 108)
I20260925 12:17:39.888656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 8098
I20260925 12:17:39.889319 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:39.888731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46165 (local address 127.0.0.1:34374)
0925 12:17:39.889236 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:39.889276 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":48,"threads_started":1}
W20260925 12:17:39.889657 7873 catalog_manager.cc:4584] TS 02eeb6fd25e64aee8e2d44453426cc4d (127.0.98.1:46165): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 328968991cf4452ca2d02d46e8179e1d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46165: connect: Connection refused (error 111)
I20260925 12:17:39.893918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 8238
I20260925 12:17:39.898178 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 7854
2026-09-25T12:17:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2885 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:17:39 dist-test-slave-clmx krb5kdc[8504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:17:39 dist-test-slave-clmx krb5kdc[8504](info): set up 2 sockets
Sep 25 12:17:39 dist-test-slave-clmx krb5kdc[8504](info): commencing operation
krb5kdc: starting...
W20260925 12:17:40.876134 392 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:17:40 dist-test-slave-clmx krb5kdc[8504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:17:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:17:40.977581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39185
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:41.051429 8517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:41.051638 8517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:41.051699 8517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:41.052779 8517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:41.052841 8517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:41.052871 8517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:41.052901 8517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:41.052925 8517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:17:41.054514 8517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:41.054744 8517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:41.054936 8517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:41.059151 8523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:41.059132 8522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:41.059125 8525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:41.059302 8517 server_base.cc:1034] running on GCE node
I20260925 12:17:41.059552 8517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:41.059829 8517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:41.060945 8517 hybrid_clock.cc:648] HybridClock initialized: now 1790338661060941 us; error 35 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T12:17:41Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1165 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:17:41 dist-test-slave-clmx krb5kdc[8530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:17:41 dist-test-slave-clmx krb5kdc[8530](info): set up 2 sockets
Sep 25 12:17:41 dist-test-slave-clmx krb5kdc[8530](info): commencing operation
krb5kdc: starting...
W20260925 12:17:42.079835 392 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:17:42 dist-test-slave-clmx krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:17:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:17:42.182188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45839
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:42.256080 8543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:42.256286 8543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:42.256320 8543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:42.257387 8543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:42.257447 8543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:42.257469 8543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:42.257495 8543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:42.257524 8543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:17:42.258860 8543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:42.259100 8543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:42.259301 8543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:42.263511 8549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:42.263530 8548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:42.263674 8543 server_base.cc:1034] running on GCE node
W20260925 12:17:42.263501 8551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:42.263994 8543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:42.264281 8543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:42.265439 8543 hybrid_clock.cc:648] HybridClock initialized: now 1790338662265425 us; error 47 us; skew 500 ppm
Sep 25 12:17:42 dist-test-slave-clmx krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:42.267839 8543 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:17:42.268239 8543 webserver.cc:460] Webserver started at http://127.0.98.62:44133/ using document root <none> and password file <none>
I20260925 12:17:42.268463 8543 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:42.268507 8543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:42.268639 8543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:42.269611 8543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "43280ac15a624ddf96ef4acee62fe338"
format_stamp: "Formatted at 2026-09-25 12:17:42 on dist-test-slave-clmx"
server_key: "ea879600debf762081be4b0ab4635000"
server_key_iv: "60a08b0de653791865b60d4902e09eb5"
server_key_version: "encryptionkey@0"
I20260925 12:17:42.269954 8543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "43280ac15a624ddf96ef4acee62fe338"
format_stamp: "Formatted at 2026-09-25 12:17:42 on dist-test-slave-clmx"
server_key: "ea879600debf762081be4b0ab4635000"
server_key_iv: "60a08b0de653791865b60d4902e09eb5"
server_key_version: "encryptionkey@0"
I20260925 12:17:42.271186 8543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:17:42.271914 8558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:42.272083 8543 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:42.272162 8543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "43280ac15a624ddf96ef4acee62fe338"
format_stamp: "Formatted at 2026-09-25 12:17:42 on dist-test-slave-clmx"
server_key: "ea879600debf762081be4b0ab4635000"
server_key_iv: "60a08b0de653791865b60d4902e09eb5"
server_key_version: "encryptionkey@0"
I20260925 12:17:42.272209 8543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:42.282280 8543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:42.283900 8543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:42.284011 8543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:42.287441 8543 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45839
I20260925 12:17:42.287492 8610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45839 every 8 connection(s)
I20260925 12:17:42.287943 8543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:42.288563 8611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:42.290438 8611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338: Bootstrap starting.
I20260925 12:17:42.290953 8611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:42.291226 8611 log.cc:824] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:42.291934 8611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338: No bootstrap required, opened a new log
I20260925 12:17:42.293378 8611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43280ac15a624ddf96ef4acee62fe338" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45839 } }
I20260925 12:17:42.293519 8611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:42.293563 8611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43280ac15a624ddf96ef4acee62fe338, State: Initialized, Role: FOLLOWER
I20260925 12:17:42.293694 8611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43280ac15a624ddf96ef4acee62fe338" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45839 } }
I20260925 12:17:42.293768 8611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:42.293810 8611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:42.293862 8611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:42.294649 8611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43280ac15a624ddf96ef4acee62fe338" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45839 } }
I20260925 12:17:42.294759 8611 leader_election.cc:302] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43280ac15a624ddf96ef4acee62fe338; no voters:
I20260925 12:17:42.294937 8611 leader_election.cc:288] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:17:42.294979 8614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:42.295152 8614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [term 1 LEADER]: Becoming Leader. State: Replica: 43280ac15a624ddf96ef4acee62fe338, State: Running, Role: LEADER
I20260925 12:17:42.295161 8611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:42.295254 8614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43280ac15a624ddf96ef4acee62fe338" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45839 } }
I20260925 12:17:42.295601 8615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43280ac15a624ddf96ef4acee62fe338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43280ac15a624ddf96ef4acee62fe338" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45839 } } }
I20260925 12:17:42.295714 8615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:42.295609 8616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43280ac15a624ddf96ef4acee62fe338. Latest consensus state: current_term: 1 leader_uuid: "43280ac15a624ddf96ef4acee62fe338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43280ac15a624ddf96ef4acee62fe338" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45839 } } }
I20260925 12:17:42.295845 8616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:42.295992 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 8543
I20260925 12:17:42.296108 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:17:42.296314 392 external_mini_cluster.cc:1428] Setting key c0adbc2af4955c0aab9461209e497a2a
W20260925 12:17:42.297119 8629 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:17:42.297173 8629 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:17:42.297223 8622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:42.297397 8622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:17:42.298768 8622 catalog_manager.cc:1347] Generated new cluster ID: 475162be57fd4133a2fa666d2f19f34e
I20260925 12:17:42.298825 8622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:17:42 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:42.306005 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:42.296903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42534 (local address 127.0.98.62:45839)
0925 12:17:42.297375 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:42.297382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:42.297403 (+ 21us) server_negotiation.cc:407] Connection header received
0925 12:17:42.297512 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:42.297518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:42.297567 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:42.297692 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:42.298408 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:42.299432 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:42.300234 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:42.300636 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:42.300781 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:42.302932 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:42.302935 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:42.302935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:42.302945 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:42.304610 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:42.304983 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:42.304986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:42.304986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:42.305071 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:42.305280 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:42.305281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:42.305281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:42.305497 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:42.305660 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:42.305731 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 12:17:42.305854 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":91,"threads_started":1}
I20260925 12:17:42.310227 8622 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:17:42.310653 8622 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:42.316258 8622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338: Generated new TSK 0
I20260925 12:17:42.316425 8622 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:17:42.343726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45839
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:42.418262 8637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:42.418491 8637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:42.418529 8637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:42.419677 8637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:42.419736 8637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:42.419778 8637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:17:42.421164 8637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:42.421413 8637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:42.421613 8637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:42.426112 8645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:42.426196 8643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:42.426195 8642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:42.426298 8637 server_base.cc:1034] running on GCE node
I20260925 12:17:42.426535 8637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:42.426756 8637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:42.427903 8637 hybrid_clock.cc:648] HybridClock initialized: now 1790338662427892 us; error 43 us; skew 500 ppm
Sep 25 12:17:42 dist-test-slave-clmx krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:42.430235 8637 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:17:42.430550 8637 webserver.cc:460] Webserver started at http://127.0.98.1:39625/ using document root <none> and password file <none>
I20260925 12:17:42.430702 8637 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:42.430737 8637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:42.430810 8637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:42.431604 8637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "f8f38b7f2db7439ead7a1ffa9d06f4ac"
format_stamp: "Formatted at 2026-09-25 12:17:42 on dist-test-slave-clmx"
server_key: "c536a9aecea50c918bf99efb81b99647"
server_key_iv: "9126e2ec4c704ab85a12472556ef3be3"
server_key_version: "encryptionkey@0"
I20260925 12:17:42.431939 8637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f8f38b7f2db7439ead7a1ffa9d06f4ac"
format_stamp: "Formatted at 2026-09-25 12:17:42 on dist-test-slave-clmx"
server_key: "c536a9aecea50c918bf99efb81b99647"
server_key_iv: "9126e2ec4c704ab85a12472556ef3be3"
server_key_version: "encryptionkey@0"
I20260925 12:17:42.433005 8637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:17:42.433643 8652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:42.433789 8637 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:17:42.433866 8637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "f8f38b7f2db7439ead7a1ffa9d06f4ac"
format_stamp: "Formatted at 2026-09-25 12:17:42 on dist-test-slave-clmx"
server_key: "c536a9aecea50c918bf99efb81b99647"
server_key_iv: "9126e2ec4c704ab85a12472556ef3be3"
server_key_version: "encryptionkey@0"
I20260925 12:17:42.433916 8637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:42.444998 8637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:42.446534 8637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:42.446627 8637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:42.446770 8637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:42.447038 8637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:42.447068 8637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:42.447088 8637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:42.447131 8637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:42.452539 8637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46143
I20260925 12:17:42.452603 8765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46143 every 8 connection(s)
I20260925 12:17:42.452872 8637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:17:42.457512 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 8637
I20260925 12:17:42.457640 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:17:42.457816 392 external_mini_cluster.cc:1428] Setting key ef1c8384e48f26bba1d3b4d1ab93bc6d
Sep 25 12:17:42 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:42.462918 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:42.453510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44605 (local address 127.0.98.62:45839)
0925 12:17:42.453568 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:42.453570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:42.453755 (+ 185us) server_negotiation.cc:407] Connection header received
0925 12:17:42.454611 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:42.454612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:42.454625 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:42.454688 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:42.455234 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:42.455778 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:42.456842 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:42.457053 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:42.457125 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:42.459537 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:42.459538 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:42.459539 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:42.459546 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:42.461208 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:42.461764 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:42.461765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:42.461766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:42.461821 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:42.462169 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:42.462170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:42.462171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:42.462308 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:42.462373 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:42.462765 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 12:17:42.462839 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 12:17:42.463385 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:42.453500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.1:44605)
0925 12:17:42.453653 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:42.453683 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:42.454506 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:42.454763 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:42.454766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:42.454989 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:42.455146 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:42.455150 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:42.455863 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:42.455864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:42.456740 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:42.456743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:42.457141 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:42.457142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:42.457246 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:42.457724 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:42.457733 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:42.459439 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:42.461311 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:42.461331 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:42.461333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:42.461671 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:42.461895 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:42.461896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:42.461896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:42.462023 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:42.462418 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:42.462424 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:42.462599 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 12:17:42.463007 (+ 408us) client_negotiation.cc:240] Negotiation successful
0925 12:17:42.463185 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260925 12:17:42.464697 8766 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45839
I20260925 12:17:42.464792 8766 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:42.465205 8766 heartbeater.cc:511] Master 127.0.98.62:45839 requested a full tablet report, sending...
I20260925 12:17:42.465883 8575 ts_manager.cc:194] Registered new tserver with Master: f8f38b7f2db7439ead7a1ffa9d06f4ac (127.0.98.1:46143)
I20260925 12:17:42.467319 8575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44605
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:17:42.490696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45839
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:42.565752 8771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:42.565920 8771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:42.565943 8771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:42.567035 8771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:42.567081 8771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:42.567131 8771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:17:42.568572 8771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:42.568804 8771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:42.569021 8771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:42.573654 8777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:42.573647 8779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:42.573657 8776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:42.573897 8771 server_base.cc:1034] running on GCE node
I20260925 12:17:42.574092 8771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:42.574342 8771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:42.575497 8771 hybrid_clock.cc:648] HybridClock initialized: now 1790338662575495 us; error 40 us; skew 500 ppm
Sep 25 12:17:42 dist-test-slave-clmx krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:42.577838 8771 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:17:42.578230 8771 webserver.cc:460] Webserver started at http://127.0.98.2:34657/ using document root <none> and password file <none>
I20260925 12:17:42.578411 8771 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:42.578449 8771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:42.578523 8771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:42.579352 8771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "e3944fa900524185b7df5c0538d0a140"
format_stamp: "Formatted at 2026-09-25 12:17:42 on dist-test-slave-clmx"
server_key: "4634819ca4d951ea357f5dff3b4a83a1"
server_key_iv: "0ed481806aef7193e26d0ccd840bf43b"
server_key_version: "encryptionkey@0"
I20260925 12:17:42.579610 8771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e3944fa900524185b7df5c0538d0a140"
format_stamp: "Formatted at 2026-09-25 12:17:42 on dist-test-slave-clmx"
server_key: "4634819ca4d951ea357f5dff3b4a83a1"
server_key_iv: "0ed481806aef7193e26d0ccd840bf43b"
server_key_version: "encryptionkey@0"
I20260925 12:17:42.580812 8771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:42.581405 8786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:42.581543 8771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:42.581624 8771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "e3944fa900524185b7df5c0538d0a140"
format_stamp: "Formatted at 2026-09-25 12:17:42 on dist-test-slave-clmx"
server_key: "4634819ca4d951ea357f5dff3b4a83a1"
server_key_iv: "0ed481806aef7193e26d0ccd840bf43b"
server_key_version: "encryptionkey@0"
I20260925 12:17:42.581666 8771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:42.616261 8771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:42.618077 8771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:42.618176 8771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:42.618345 8771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:42.618563 8771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:42.618590 8771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:42.618609 8771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:42.618623 8771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:42.624006 8771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43879
I20260925 12:17:42.624063 8899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43879 every 8 connection(s)
I20260925 12:17:42.624311 8771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:17:42.624696 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 8771
I20260925 12:17:42.624815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:17:42.625027 392 external_mini_cluster.cc:1428] Setting key 6c1eabb68ef37bc01f5577d51160a98b
Sep 25 12:17:42 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 12:17:42.634155 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:42.624976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36831 (local address 127.0.98.62:45839)
0925 12:17:42.625032 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:42.625034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:42.625280 (+ 246us) server_negotiation.cc:407] Connection header received
0925 12:17:42.626498 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:42.626499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:42.626508 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:42.626557 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:42.627361 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:42.628162 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:42.629080 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:42.629412 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:42.629487 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:42.631383 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:42.631385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:42.631385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:42.631393 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:42.632927 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:42.633305 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:42.633307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:42.633307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:42.633376 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:42.633633 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:42.633634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:42.633635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:42.633769 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:42.633824 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:42.634036 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 12:17:42.634095 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:17:42.634341 8902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:42.624962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.2:36831)
0925 12:17:42.625131 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:42.625177 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:42.626399 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:42.626622 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:42.626626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:42.626998 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:42.627260 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:42.627266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:42.628254 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:42.628256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:42.628998 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:42.629000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:42.629489 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:42.629490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:42.629586 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:42.630044 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:42.630051 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:42.631265 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:42.633015 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:42.633020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:42.633021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:42.633234 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:42.633459 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:42.633460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:42.633460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:42.633565 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:42.633831 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:42.633834 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:42.633932 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 12:17:42.634008 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 12:17:42.634112 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260925 12:17:42.634786 8900 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45839
I20260925 12:17:42.634863 8900 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:42.634974 8900 heartbeater.cc:511] Master 127.0.98.62:45839 requested a full tablet report, sending...
I20260925 12:17:42.635246 8575 ts_manager.cc:194] Registered new tserver with Master: e3944fa900524185b7df5c0538d0a140 (127.0.98.2:43879)
I20260925 12:17:42.635682 8575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:17:42.648283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45839
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:42.722131 8905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:42.722291 8905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:42.722313 8905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:42.723340 8905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:42.723372 8905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:42.723397 8905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:17:42.724772 8905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:42.724937 8905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:42.725081 8905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:42.729429 8913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:42.729457 8911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:42.729450 8910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:42.729642 8905 server_base.cc:1034] running on GCE node
I20260925 12:17:42.729828 8905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:42.730067 8905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:42.731240 8905 hybrid_clock.cc:648] HybridClock initialized: now 1790338662731227 us; error 21 us; skew 500 ppm
Sep 25 12:17:42 dist-test-slave-clmx krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:42.733673 8905 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:17:42.734079 8905 webserver.cc:460] Webserver started at http://127.0.98.3:37059/ using document root <none> and password file <none>
I20260925 12:17:42.734299 8905 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:42.734380 8905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:42.734526 8905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:42.735466 8905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "2215fe60c8c0442c8f9f5dad5c1afcce"
format_stamp: "Formatted at 2026-09-25 12:17:42 on dist-test-slave-clmx"
server_key: "da83bcb7b986e1e96ed683a179d31033"
server_key_iv: "1b51a09c8b400b36c9f6d714d7de543b"
server_key_version: "encryptionkey@0"
I20260925 12:17:42.735834 8905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2215fe60c8c0442c8f9f5dad5c1afcce"
format_stamp: "Formatted at 2026-09-25 12:17:42 on dist-test-slave-clmx"
server_key: "da83bcb7b986e1e96ed683a179d31033"
server_key_iv: "1b51a09c8b400b36c9f6d714d7de543b"
server_key_version: "encryptionkey@0"
I20260925 12:17:42.737252 8905 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:17:42.737967 8920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:42.738119 8905 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:17:42.738229 8905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "2215fe60c8c0442c8f9f5dad5c1afcce"
format_stamp: "Formatted at 2026-09-25 12:17:42 on dist-test-slave-clmx"
server_key: "da83bcb7b986e1e96ed683a179d31033"
server_key_iv: "1b51a09c8b400b36c9f6d714d7de543b"
server_key_version: "encryptionkey@0"
I20260925 12:17:42.738299 8905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:42.749648 8905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:42.751884 8905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:42.752030 8905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:42.752261 8905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:42.752549 8905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:42.752600 8905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:42.752653 8905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:42.752681 8905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:42.758597 8905 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37809
I20260925 12:17:42.758631 9033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37809 every 8 connection(s)
I20260925 12:17:42.758899 8905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:17:42.762002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 8905
I20260925 12:17:42.762111 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:17:42.762288 392 external_mini_cluster.cc:1428] Setting key f0a9969d93accbc344fca98b53f93a19
Sep 25 12:17:42 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:17:42.767562 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:42.759506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43049 (local address 127.0.98.62:45839)
0925 12:17:42.759551 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:42.759553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:42.759827 (+ 274us) server_negotiation.cc:407] Connection header received
0925 12:17:42.760628 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:42.760629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:42.760638 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:42.760687 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:42.761187 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:42.761642 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:42.762989 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:42.763257 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:42.763315 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:42.765167 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:42.765168 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:42.765168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:42.765174 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:42.766565 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:42.766879 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:42.766879 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:42.766880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:42.766927 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:42.767119 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:42.767119 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:42.767119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:42.767211 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:42.767258 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:42.767450 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 12:17:42.767504 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:17:42.767742 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:42.759494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.3:43049)
0925 12:17:42.759677 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:42.759714 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:42.760549 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:42.760767 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:42.760770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:42.760974 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:42.761125 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:42.761128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:42.761711 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:42.761711 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:42.762909 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:42.762911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:42.763333 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:42.763334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:42.763430 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:42.763873 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:42.763880 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:42.765091 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:42.766631 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:42.766635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:42.766636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:42.766819 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:42.766984 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:17:42.766984 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:17:42.766985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:17:42.767067 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:17:42.767262 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:42.767265 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:42.767364 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 12:17:42.767438 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 12:17:42.767536 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":79,"threads_started":1}
I20260925 12:17:42.768200 9034 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45839
I20260925 12:17:42.768278 9034 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:42.768397 9034 heartbeater.cc:511] Master 127.0.98.62:45839 requested a full tablet report, sending...
I20260925 12:17:42.768749 8575 ts_manager.cc:194] Registered new tserver with Master: 2215fe60c8c0442c8f9f5dad5c1afcce (127.0.98.3:37809)
I20260925 12:17:42.769129 8575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43049
I20260925 12:17:42.773867 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 12:17:42.833573 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 8637
I20260925 12:17:42.837891 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 8771
I20260925 12:17:42.841742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 8905
I20260925 12:17:42.845638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 8543
I20260925 12:17:42.850232 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45839
--webserver_interface=127.0.98.62
--webserver_port=44133
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:17:42.923450 9043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:42.923707 9043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:42.923753 9043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:42.924892 9043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:42.924930 9043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:42.924945 9043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:42.924994 9043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:42.925022 9043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:17:42.926442 9043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:42.926663 9043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:42.926872 9043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:42.931267 9048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:42.931377 9043 server_base.cc:1034] running on GCE node
W20260925 12:17:42.931293 9049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:42.931337 9051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:42.931751 9043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:42.931989 9043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:42.933125 9043 hybrid_clock.cc:648] HybridClock initialized: now 1790338662933108 us; error 32 us; skew 500 ppm
Sep 25 12:17:42 dist-test-slave-clmx krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:42.935398 9043 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:17:42.935860 9043 webserver.cc:460] Webserver started at http://127.0.98.62:44133/ using document root <none> and password file <none>
I20260925 12:17:42.936065 9043 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:42.936112 9043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:42.937317 9043 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:42.937909 9058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:42.938059 9043 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:42.938148 9043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "43280ac15a624ddf96ef4acee62fe338"
format_stamp: "Formatted at 2026-09-25 12:17:42 on dist-test-slave-clmx"
server_key: "ea879600debf762081be4b0ab4635000"
server_key_iv: "60a08b0de653791865b60d4902e09eb5"
server_key_version: "encryptionkey@0"
I20260925 12:17:42.938427 9043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:42.951872 9043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:42.953356 9043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:42.953500 9043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:42.957270 9043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45839
I20260925 12:17:42.957361 9110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45839 every 8 connection(s)
I20260925 12:17:42.957765 9043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:42.958789 9111 sys_catalog.cc:263] Verifying existing consensus state
I20260925 12:17:42.959293 9111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338: Bootstrap starting.
I20260925 12:17:42.964022 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 9043
I20260925 12:17:42.964293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46143
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45839
--webserver_port=39625
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:17:42.966919 9111 log.cc:824] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:42.968477 9111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:17:42.968729 9111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338: Bootstrap complete.
I20260925 12:17:42.970716 9111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43280ac15a624ddf96ef4acee62fe338" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45839 } }
I20260925 12:17:42.970955 9111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43280ac15a624ddf96ef4acee62fe338, State: Initialized, Role: FOLLOWER
I20260925 12:17:42.971159 9111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43280ac15a624ddf96ef4acee62fe338" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45839 } }
I20260925 12:17:42.971251 9111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:42.971300 9111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:42.971346 9111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [term 1 FOLLOWER]: Advancing to term 2
I20260925 12:17:42.972252 9111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43280ac15a624ddf96ef4acee62fe338" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45839 } }
I20260925 12:17:42.972364 9111 leader_election.cc:302] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43280ac15a624ddf96ef4acee62fe338; no voters:
I20260925 12:17:42.972548 9111 leader_election.cc:288] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 12:17:42.972601 9115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 12:17:42.972731 9115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [term 2 LEADER]: Becoming Leader. State: Replica: 43280ac15a624ddf96ef4acee62fe338, State: Running, Role: LEADER
I20260925 12:17:42.972779 9111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:42.972827 9115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43280ac15a624ddf96ef4acee62fe338" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45839 } }
I20260925 12:17:42.973078 9115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43280ac15a624ddf96ef4acee62fe338. Latest consensus state: current_term: 2 leader_uuid: "43280ac15a624ddf96ef4acee62fe338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43280ac15a624ddf96ef4acee62fe338" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45839 } } }
I20260925 12:17:42.973281 9115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:42.973075 9116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "43280ac15a624ddf96ef4acee62fe338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43280ac15a624ddf96ef4acee62fe338" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45839 } } }
I20260925 12:17:42.973578 9116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:42.973600 9125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:42.974056 9125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:17:42.974182 9125 catalog_manager.cc:1259] Loaded cluster ID: 475162be57fd4133a2fa666d2f19f34e
I20260925 12:17:42.974413 9125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:17:42.975147 9125 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:42.975255 9125 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 43280ac15a624ddf96ef4acee62fe338: Loaded TSK: 0
I20260925 12:17:42.975385 9125 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 12:17:43.054430 9113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:43.054600 9113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:43.054620 9113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:43.055727 9113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:43.055781 9113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:43.055826 9113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:17:43.057204 9113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39625
--tserver_master_addrs=127.0.98.62:45839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:43.057471 9113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:43.057687 9113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:43.062175 9138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:43.062180 9137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:43.062189 9140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:43.062407 9113 server_base.cc:1034] running on GCE node
I20260925 12:17:43.062688 9113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:43.062913 9113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:43.064064 9113 hybrid_clock.cc:648] HybridClock initialized: now 1790338663064038 us; error 44 us; skew 500 ppm
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338663, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:43.066349 9113 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 12:17:43.066730 9113 webserver.cc:460] Webserver started at http://127.0.98.1:39625/ using document root <none> and password file <none>
I20260925 12:17:43.066941 9113 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:43.066990 9113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:43.068346 9113 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:43.068908 9147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:43.069078 9113 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:43.069170 9113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "f8f38b7f2db7439ead7a1ffa9d06f4ac"
format_stamp: "Formatted at 2026-09-25 12:17:42 on dist-test-slave-clmx"
server_key: "c536a9aecea50c918bf99efb81b99647"
server_key_iv: "9126e2ec4c704ab85a12472556ef3be3"
server_key_version: "encryptionkey@0"
I20260925 12:17:43.069489 9113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:43.089962 9113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:43.092003 9113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:43.092149 9113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:43.092374 9113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:43.092686 9113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:43.092746 9113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:43.092785 9113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:43.092803 9113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:43.099793 9113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46143
I20260925 12:17:43.099858 9260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46143 every 8 connection(s)
I20260925 12:17:43.100168 9113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:43.108265 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:43.100778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.1:50419)
0925 12:17:43.100954 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:43.100988 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:43.101490 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:43.101877 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:43.101881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:43.102587 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:43.102810 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:43.102815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.103903 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.103904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:43.105151 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:43.105153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.105676 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.105676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:43.105767 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:43.106218 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:43.106225 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:43.108148 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
W20260925 12:17:43.108397 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:43.100880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50419 (local address 127.0.98.62:45839)
0925 12:17:43.101059 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:43.101067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:43.101100 (+ 33us) server_negotiation.cc:407] Connection header received
0925 12:17:43.101674 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:43.101678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:43.101714 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:43.101808 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:43.102908 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.103814 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.105225 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.105591 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.105676 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:43.108250 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50419: BlockingRecv error: recv got EOF from 127.0.98.1:50419 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":61,"threads_started":1}
W20260925 12:17:43.108496 9261 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45839 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45839: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260925 12:17:43.108839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 9113
I20260925 12:17:43.109139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43879
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45839
--webserver_port=34657
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:43.112834 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:43.108711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.1:51433)
0925 12:17:43.108907 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:43.108917 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:43.108964 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:43.109111 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:43.109112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:43.109251 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:43.109301 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:43.109303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.109851 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.109852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:43.110482 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:43.110484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.110896 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.110896 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:43.110936 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:43.111271 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:43.111275 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:43.112787 (+ 1512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166}
W20260925 12:17:43.113031 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:43.108771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51433 (local address 127.0.98.62:45839)
0925 12:17:43.108837 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:43.108839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:43.109014 (+ 175us) server_negotiation.cc:407] Connection header received
0925 12:17:43.109030 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:43.109031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:43.109040 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:43.109108 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:43.109380 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.109778 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.110555 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.110822 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.110900 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:43.112898 (+ 1998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51433: BlockingRecv error: recv got EOF from 127.0.98.1:51433 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:43.117147 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:43.113048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.1:58405)
0925 12:17:43.113104 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:43.113113 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:43.113168 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:43.113355 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:43.113355 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:43.113455 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:43.113477 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:43.113478 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.114382 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.114383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:43.114807 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:43.114808 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.115274 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.115275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:43.115306 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:43.115689 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:43.115693 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:43.117112 (+ 1419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 12:17:43.117285 9261 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 12:17:43.117394 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:43.113106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58405 (local address 127.0.98.62:45839)
0925 12:17:43.113239 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:43.113241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:43.113249 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:43.113268 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:43.113269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:43.113279 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:43.113345 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:43.113550 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.114297 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.114889 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.115197 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.115264 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:43.117212 (+ 1948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58405: BlockingRecv error: recv got EOF from 127.0.98.1:58405 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":94,"threads_started":1}
W20260925 12:17:43.186707 9265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:43.186877 9265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:43.186898 9265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:43.188083 9265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:43.188120 9265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:43.188146 9265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:17:43.189522 9265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34657
--tserver_master_addrs=127.0.98.62:45839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:43.189785 9265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:43.189991 9265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:43.194658 9271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:43.194664 9272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:43.194687 9274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:43.194677 9265 server_base.cc:1034] running on GCE node
I20260925 12:17:43.195026 9265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:43.195209 9265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:43.196367 9265 hybrid_clock.cc:648] HybridClock initialized: now 1790338663196355 us; error 45 us; skew 500 ppm
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338663, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:43.198552 9265 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 12:17:43.199054 9265 webserver.cc:460] Webserver started at http://127.0.98.2:34657/ using document root <none> and password file <none>
I20260925 12:17:43.199285 9265 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:43.199365 9265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:43.200644 9265 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:43.201157 9281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:43.201334 9265 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:43.201426 9265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "e3944fa900524185b7df5c0538d0a140"
format_stamp: "Formatted at 2026-09-25 12:17:42 on dist-test-slave-clmx"
server_key: "4634819ca4d951ea357f5dff3b4a83a1"
server_key_iv: "0ed481806aef7193e26d0ccd840bf43b"
server_key_version: "encryptionkey@0"
I20260925 12:17:43.201718 9265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:43.221541 9265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:43.223078 9265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:43.223232 9265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:43.223439 9265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:43.223784 9265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:43.223838 9265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:43.223871 9265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:43.223898 9265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:43.229260 9265 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43879
I20260925 12:17:43.229336 9394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43879 every 8 connection(s)
I20260925 12:17:43.229625 9265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:17:43.233806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 9265
I20260925 12:17:43.234081 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37809
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45839
--webserver_port=37059
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 12:17:43.236092 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:43.230242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34553 (local address 127.0.98.62:45839)
0925 12:17:43.230281 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:43.230283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:43.230445 (+ 162us) server_negotiation.cc:407] Connection header received
0925 12:17:43.230838 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:43.230839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:43.230850 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:43.230903 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:43.231678 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.232125 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.232965 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.233132 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.233198 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:43.236039 (+ 2841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34553: BlockingRecv error: recv got EOF from 127.0.98.2:34553 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:17:43.236142 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:43.230204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.2:34553)
0925 12:17:43.230352 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:43.230386 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:43.230763 (+ 377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:43.230947 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:43.230949 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:43.231422 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:43.231572 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:43.231575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.232190 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.232191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:43.232876 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:43.232878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.233214 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.233214 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:43.233306 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:43.233697 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:43.233704 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:43.235977 (+ 2273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
W20260925 12:17:43.236322 9395 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45839 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45839: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:43.240471 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:43.236533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.2:36303)
0925 12:17:43.236581 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:43.236590 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:43.236648 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:43.236757 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:43.236757 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:43.236879 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:43.236915 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:43.236916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.237598 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.237599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:43.238074 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:43.238076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.238418 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.238418 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:43.238491 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:43.238868 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:43.238873 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:43.240406 (+ 1533us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 12:17:43.240507 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:43.236592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36303 (local address 127.0.98.62:45839)
0925 12:17:43.236647 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:43.236650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:43.236657 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:17:43.236686 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:43.236687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:43.236695 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:43.236756 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:43.236988 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.237513 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.238164 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.238351 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.238420 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:43.240457 (+ 2037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36303: BlockingRecv error: recv got EOF from 127.0.98.2:36303 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:43.244594 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:43.240731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.2:42917)
0925 12:17:43.240765 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:43.240773 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:43.240803 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:43.240907 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:43.240907 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:43.241019 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:43.241050 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:43.241051 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.241695 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.241697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:43.242191 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:43.242193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.242523 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.242524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:43.242569 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:43.242939 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:43.242943 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:43.244525 (+ 1582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 12:17:43.244642 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:43.240777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42917 (local address 127.0.98.62:45839)
0925 12:17:43.240816 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:43.240818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:43.240825 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:17:43.240844 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:43.240845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:43.240853 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:43.240894 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:43.241132 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.241610 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.242273 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.242449 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.242521 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:43.244585 (+ 2064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42917: BlockingRecv error: recv got EOF from 127.0.98.2:42917 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 12:17:43.244693 9395 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 12:17:43.311372 9398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:43.311582 9398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:43.311617 9398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:43.312750 9398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:43.312821 9398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:43.312868 9398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:17:43.314283 9398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37059
--tserver_master_addrs=127.0.98.62:45839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:43.314535 9398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:43.314762 9398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:43.319649 9406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:43.319662 9398 server_base.cc:1034] running on GCE node
W20260925 12:17:43.319615 9404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:43.319602 9403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:43.320043 9398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:43.320235 9398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:43.321363 9398 hybrid_clock.cc:648] HybridClock initialized: now 1790338663321352 us; error 30 us; skew 500 ppm
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338663, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:17:43.323542 9398 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 12:17:43.323997 9398 webserver.cc:460] Webserver started at http://127.0.98.3:37059/ using document root <none> and password file <none>
I20260925 12:17:43.324162 9398 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:43.324209 9398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:43.325362 9398 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:17:43.325821 9413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:43.326012 9398 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:43.326076 9398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "2215fe60c8c0442c8f9f5dad5c1afcce"
format_stamp: "Formatted at 2026-09-25 12:17:42 on dist-test-slave-clmx"
server_key: "da83bcb7b986e1e96ed683a179d31033"
server_key_iv: "1b51a09c8b400b36c9f6d714d7de543b"
server_key_version: "encryptionkey@0"
I20260925 12:17:43.326382 9398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:43.341609 9398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:43.343499 9398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:43.343695 9398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:43.343899 9398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:43.344175 9398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:43.344228 9398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:43.344267 9398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:43.344295 9398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:43.349359 9398 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37809
I20260925 12:17:43.349409 9526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37809 every 8 connection(s)
I20260925 12:17:43.349723 9398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:43.356045 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:43.350281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.3:58603)
0925 12:17:43.350395 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:43.350430 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:43.350895 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:43.351114 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:43.351117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:43.351622 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:43.351856 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:43.351860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.352469 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.352470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:43.353219 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:43.353221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.353617 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.353618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:43.353729 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:43.354087 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:43.354093 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:43.355932 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":55,"threads_started":1}
W20260925 12:17:43.356052 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:43.350307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58603 (local address 127.0.98.62:45839)
0925 12:17:43.350356 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:43.350358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:43.350504 (+ 146us) server_negotiation.cc:407] Connection header received
0925 12:17:43.350990 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:43.350992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:43.351004 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:43.351050 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:43.351947 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.352387 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.353312 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.353525 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.353608 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:43.356009 (+ 2401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58603: BlockingRecv error: recv got EOF from 127.0.98.3:58603 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 12:17:43.356341 9527 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45839 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45839: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260925 12:17:43.358824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 9398
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:43.360728 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:43.356533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.3:56003)
0925 12:17:43.356578 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:43.356589 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:43.356628 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:43.356861 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:43.356862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:43.356979 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:43.357013 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:43.357014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.357587 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.357588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:43.358160 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:43.358162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.358497 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.358497 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:43.358563 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:43.358980 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:43.358984 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:43.360677 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 12:17:43.360811 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:43.356720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56003 (local address 127.0.98.62:45839)
0925 12:17:43.356765 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:43.356766 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:43.356772 (+ 6us) server_negotiation.cc:407] Connection header received
0925 12:17:43.356788 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:43.356789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:43.356796 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:43.356858 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:43.357085 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.357513 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.358246 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.358420 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.358496 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:43.360742 (+ 2246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56003: BlockingRecv error: recv got EOF from 127.0.98.3:56003 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:43 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:43.364872 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:43.360964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.3:52183)
0925 12:17:43.361006 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:43.361016 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:43.361049 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:43.361148 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:43.361148 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:43.361268 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:43.361292 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:43.361293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.361836 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.361837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:43.362426 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:43.362428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.362706 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.362706 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:43.362759 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:43.363035 (+ 276us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:43.363040 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:43.364810 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 12:17:43.364989 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:43.361006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52183 (local address 127.0.98.62:45839)
0925 12:17:43.361036 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:43.361037 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:43.361050 (+ 13us) server_negotiation.cc:407] Connection header received
0925 12:17:43.361097 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:43.361097 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:43.361103 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:43.361144 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:43.361365 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.361761 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.362496 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.362643 (+ 147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.362688 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:43.364930 (+ 2242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52183: BlockingRecv error: recv got EOF from 127.0.98.3:52183 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 12:17:43.364990 9527 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 12:17:43.366214 9266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:43.359222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52456 (local address 127.0.98.62:45839)
0925 12:17:43.359284 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:43.359286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:43.359396 (+ 110us) server_negotiation.cc:407] Connection header received
0925 12:17:43.359476 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:43.359477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:43.359487 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:43.359546 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:43.360224 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.360774 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.361762 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:43.361980 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:43.362022 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:43.363263 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:43.363267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:43.363267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:17:43.363275 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:43.365183 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:43.365459 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:43.365461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:43.365461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:43.365535 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:17:43.365675 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:17:43.365675 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:17:43.365676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:17:43.365836 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:43.365925 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:43.366001 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 12:17:43.366081 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 12:17:44 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:44 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:44.129081 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:44.117671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.1:40125)
0925 12:17:44.117781 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:44.117794 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:44.117855 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:44.117987 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:44.117988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:44.118131 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:44.118178 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:44.118179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:44.118739 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:44.118739 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:44.119224 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:44.119226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:44.119509 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:44.119509 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:44.119545 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:44.127097 (+ 7552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:44.127102 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:44.129022 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":45,"threads_started":1}
W20260925 12:17:44.129395 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:44.117755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40125 (local address 127.0.98.62:45839)
0925 12:17:44.117882 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:44.117883 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:44.117890 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:17:44.117909 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:44.117910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:44.117918 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:44.117970 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:44.118240 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:44.118675 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:44.119291 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:44.119452 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:44.119495 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:44.129338 (+ 9843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40125: BlockingRecv error: recv got EOF from 127.0.98.1:40125 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
Sep 25 12:17:44 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:44 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:44.249012 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:44.244989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.2:60239)
0925 12:17:44.245098 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:44.245109 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:44.245178 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:44.245290 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:44.245291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:44.245435 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:44.245478 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:44.245480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:44.246058 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:44.246058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:44.246554 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:44.246556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:44.246841 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:44.246842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:44.246878 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:44.247256 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:44.247260 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:44.248899 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":69,"threads_started":1}
W20260925 12:17:44.249049 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:44.245052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60239 (local address 127.0.98.62:45839)
0925 12:17:44.245100 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:44.245102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:44.245167 (+ 65us) server_negotiation.cc:407] Connection header received
0925 12:17:44.245228 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:44.245228 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:44.245237 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:44.245275 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:44.245537 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:44.245997 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:44.246617 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:44.246784 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:44.246830 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:44.248983 (+ 2153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60239: BlockingRecv error: recv got EOF from 127.0.98.2:60239 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 25 12:17:44 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:44 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:44.369377 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:44.365322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.3:48655)
0925 12:17:44.365450 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:44.365460 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:44.365516 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:44.365641 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:44.365642 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:44.365789 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:44.365841 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:44.365842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:44.366374 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:44.366375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:44.366847 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:44.366848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:44.367246 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:44.367247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:44.367313 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:44.367782 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:44.367787 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:44.369276 (+ 1489us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
W20260925 12:17:44.369397 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:44.365419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48655 (local address 127.0.98.62:45839)
0925 12:17:44.365460 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:44.365462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:44.365501 (+ 39us) server_negotiation.cc:407] Connection header received
0925 12:17:44.365568 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:44.365568 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:44.365579 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:44.365623 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:44.365894 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:44.366320 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:44.366941 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:44.367167 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:44.367246 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:44.369340 (+ 2094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48655: BlockingRecv error: recv got EOF from 127.0.98.3:48655 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 25 12:17:45 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:45 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:45.134501 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:45.129558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.1:39387)
0925 12:17:45.129746 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:45.129759 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:45.129855 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:45.130041 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:45.130042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:45.130206 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:45.130252 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:45.130254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:45.130922 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:45.130923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:45.131472 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:45.131473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:45.131909 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:45.131910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:45.131975 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:45.132475 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:45.132480 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:45.134432 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":119,"threads_started":1}
W20260925 12:17:45.134644 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:45.129625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39387 (local address 127.0.98.62:45839)
0925 12:17:45.129759 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:45.129761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:45.129828 (+ 67us) server_negotiation.cc:407] Connection header received
0925 12:17:45.129926 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:45.129928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:45.129941 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:45.130021 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:45.130331 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:45.130835 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:45.131566 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:45.131820 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:45.131908 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:45.134522 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39387: BlockingRecv error: recv got EOF from 127.0.98.1:39387 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
Sep 25 12:17:45 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:45 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:45.253885 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:45.249682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.2:56087)
0925 12:17:45.249795 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:45.249806 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:45.249895 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:45.250032 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:45.250033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:45.250207 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:45.250251 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:45.250253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:45.250841 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:45.250842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:45.251332 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:45.251333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:45.251671 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:45.251672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:45.251724 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:45.252140 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:45.252144 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:45.253833 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
W20260925 12:17:45.254015 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:45.249745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56087 (local address 127.0.98.62:45839)
0925 12:17:45.249792 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:45.249794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:45.249876 (+ 82us) server_negotiation.cc:407] Connection header received
0925 12:17:45.249949 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:45.249950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:45.249962 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:45.250010 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:45.250313 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:45.250774 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:45.251397 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:45.251564 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:45.251626 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:45.253936 (+ 2310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56087: BlockingRecv error: recv got EOF from 127.0.98.2:56087 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 25 12:17:45 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:45 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:45.373987 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:45.369763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.3:46113)
0925 12:17:45.369891 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:45.369901 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:45.369949 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:45.370068 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:45.370068 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:45.370217 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:45.370260 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:45.370262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:45.370880 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:45.370881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:45.371384 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:45.371386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:45.371719 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:45.371719 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:45.371757 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:45.372152 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:45.372156 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:45.373933 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
W20260925 12:17:45.374089 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:45.369895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46113 (local address 127.0.98.62:45839)
0925 12:17:45.369950 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:45.369951 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:45.369983 (+ 32us) server_negotiation.cc:407] Connection header received
0925 12:17:45.370001 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:45.370002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:45.370013 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:45.370049 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:45.370314 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:45.370794 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:45.371442 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:45.371650 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:45.371717 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:45.374021 (+ 2304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46113: BlockingRecv error: recv got EOF from 127.0.98.3:46113 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 12:17:46 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:46 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:46.138985 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:46.134939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.1:36995)
0925 12:17:46.135106 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:46.135116 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:46.135170 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:46.135285 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:46.135286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:46.135437 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:46.135484 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:46.135485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:46.136134 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:46.136134 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:46.136638 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:46.136639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:46.136987 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:46.136988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:46.137023 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:46.137424 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:46.137428 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:46.138927 (+ 1499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":103,"threads_started":1}
W20260925 12:17:46.139106 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:46.135002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36995 (local address 127.0.98.62:45839)
0925 12:17:46.135115 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:46.135117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:46.135175 (+ 58us) server_negotiation.cc:407] Connection header received
0925 12:17:46.135215 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:46.135215 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:46.135226 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:46.135271 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:46.135542 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:46.136068 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:46.136711 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:46.136916 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:46.136964 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:46.139007 (+ 2043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36995: BlockingRecv error: recv got EOF from 127.0.98.1:36995 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":51,"threads_started":1}
Sep 25 12:17:46 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:46 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:46.258311 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:46.254330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.2:34933)
0925 12:17:46.254480 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:46.254488 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:46.254538 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:46.254676 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:46.254677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:46.254818 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:46.254863 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:46.254864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:46.255444 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:46.255445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:46.255940 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:46.255941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:46.256227 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:46.256228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:46.256261 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:46.256682 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:46.256686 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:46.258258 (+ 1572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
W20260925 12:17:46.258440 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:46.254393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34933 (local address 127.0.98.62:45839)
0925 12:17:46.254460 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:46.254462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:46.254543 (+ 81us) server_negotiation.cc:407] Connection header received
0925 12:17:46.254595 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:46.254596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:46.254606 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:46.254658 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:46.254919 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:46.255385 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:46.255999 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:46.256172 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:46.256217 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:46.258367 (+ 2150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34933: BlockingRecv error: recv got EOF from 127.0.98.2:34933 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 12:17:46 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:46 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:46.378942 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:46.374430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.3:43513)
0925 12:17:46.374578 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:46.374600 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:46.374675 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:46.374847 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:46.374848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:46.375009 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:46.375052 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:46.375054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:46.375724 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:46.375725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:46.376262 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:46.376264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:46.376594 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:46.376595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:46.376634 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:46.377069 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:46.377073 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:46.378851 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
W20260925 12:17:46.378975 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:46.374479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43513 (local address 127.0.98.62:45839)
0925 12:17:46.374542 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:46.374544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:46.374657 (+ 113us) server_negotiation.cc:407] Connection header received
0925 12:17:46.374743 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:46.374744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:46.374764 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:46.374838 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:46.375129 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:46.375600 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:46.376333 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:46.376495 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:46.376560 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:46.378916 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43513: BlockingRecv error: recv got EOF from 127.0.98.3:43513 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 12:17:47 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:47 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:47.144160 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:47.139745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.1:54315)
0925 12:17:47.139886 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:47.139897 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:47.139982 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:47.140159 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:47.140160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:47.140319 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:47.140371 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:47.140373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:47.141115 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:47.141116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:47.141689 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:47.141691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:47.142019 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:47.142020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:47.142068 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:47.142501 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:47.142505 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:47.144106 (+ 1601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
W20260925 12:17:47.144289 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:47.139781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54315 (local address 127.0.98.62:45839)
0925 12:17:47.139919 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:47.139921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:47.139948 (+ 27us) server_negotiation.cc:407] Connection header received
0925 12:17:47.140058 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:47.140059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:47.140073 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:47.140141 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:47.140476 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:47.141035 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:47.141756 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:47.141939 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:47.141993 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:47.144189 (+ 2196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54315: BlockingRecv error: recv got EOF from 127.0.98.1:54315 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":44,"threads_started":1}
Sep 25 12:17:47 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:47 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:47.263005 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:47.258757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.2:41739)
0925 12:17:47.258916 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:47.258925 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:47.258975 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:47.259091 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:47.259091 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:47.259234 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:47.259276 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:47.259278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:47.259932 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:47.259932 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:47.260417 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:47.260418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:47.260745 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:47.260746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:47.260779 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:47.261187 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:47.261191 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:47.262922 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
W20260925 12:17:47.263067 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:47.258801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41739 (local address 127.0.98.62:45839)
0925 12:17:47.258859 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:47.258860 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:47.258979 (+ 119us) server_negotiation.cc:407] Connection header received
0925 12:17:47.259023 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:47.259024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:47.259035 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:47.259076 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:47.259331 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:47.259851 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:47.260484 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:47.260673 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:47.260740 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:47.263007 (+ 2267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41739: BlockingRecv error: recv got EOF from 127.0.98.2:41739 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 12:17:47 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:47 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:47.383476 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:47.379439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.3:52501)
0925 12:17:47.379650 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:47.379662 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:47.379729 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:47.379871 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:47.379871 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:47.380014 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:47.380058 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:47.380060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:47.380668 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:47.380668 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:47.381135 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:47.381136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:47.381424 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:47.381425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:47.381460 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:47.381883 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:47.381887 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:47.383422 (+ 1535us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":77,"threads_started":1}
W20260925 12:17:47.383589 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:47.379611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52501 (local address 127.0.98.62:45839)
0925 12:17:47.379701 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:47.379702 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:47.379710 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:47.379779 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:47.379780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:47.379792 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:47.379841 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:47.380113 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:47.380610 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:47.381194 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:47.381368 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:47.381414 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:47.383522 (+ 2108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52501: BlockingRecv error: recv got EOF from 127.0.98.3:52501 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 12:17:48 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:48 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:48.148622 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:48.144605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.1:39315)
0925 12:17:48.144746 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:48.144758 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:48.144815 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:48.144947 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:48.144948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:48.145088 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:48.145147 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:48.145148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:48.145731 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:48.145732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:48.146225 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:48.146226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:48.146526 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:48.146527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:48.146568 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:48.146963 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:48.146967 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:48.148569 (+ 1602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
W20260925 12:17:48.148731 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:48.144659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39315 (local address 127.0.98.62:45839)
0925 12:17:48.144796 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:48.144797 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:48.144807 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:17:48.144864 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:48.144865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:48.144874 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:48.144930 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:48.145205 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:48.145673 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:48.146288 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:48.146465 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:48.146511 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:48.148654 (+ 2143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39315: BlockingRecv error: recv got EOF from 127.0.98.1:39315 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
Sep 25 12:17:48 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:48 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:48.268042 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:48.263427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.2:45417)
0925 12:17:48.263580 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:48.263593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:48.263695 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:48.263915 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:48.263916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:48.264098 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:48.264143 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:48.264144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:48.264783 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:48.264784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:48.265301 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:48.265303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:48.265708 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:48.265709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:48.265772 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:48.266187 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:48.266191 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:48.267988 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":56,"threads_started":1}
W20260925 12:17:48.268174 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:48.263705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45417 (local address 127.0.98.62:45839)
0925 12:17:48.263775 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:48.263777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:48.263785 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:48.263814 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:48.263815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:48.263824 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:48.263907 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:48.264223 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:48.264703 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:48.265409 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:48.265605 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:48.265694 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:48.268092 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45417: BlockingRecv error: recv got EOF from 127.0.98.2:45417 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 12:17:48 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:48 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:48.388453 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:48.383942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.3:54081)
0925 12:17:48.384115 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:48.384128 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:48.384219 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:48.384377 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:48.384378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:48.384537 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:48.384579 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:48.384581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:48.385266 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:48.385277 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:48.385819 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:48.385820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:48.386176 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:48.386177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:48.386241 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:48.386636 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:48.386640 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:48.388341 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":81,"threads_started":1}
W20260925 12:17:48.388461 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:48.384050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54081 (local address 127.0.98.62:45839)
0925 12:17:48.384117 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:48.384119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:48.384201 (+ 82us) server_negotiation.cc:407] Connection header received
0925 12:17:48.384272 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:48.384274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:48.384288 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:48.384361 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:48.384657 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:48.385165 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:48.385906 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:48.386091 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:48.386174 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:48.388409 (+ 2235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54081: BlockingRecv error: recv got EOF from 127.0.98.3:54081 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 12:17:49 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:49 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:49.157230 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:49.149086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.1:33927)
0925 12:17:49.149242 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:49.149252 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:49.149301 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:49.149431 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:49.149431 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:49.149582 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:49.149627 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:49.149628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:49.150583 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:49.150584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:49.151148 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:49.151151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:49.151458 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:49.151459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:49.151498 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:49.155403 (+ 3905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:49.155408 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:49.157168 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
W20260925 12:17:49.157354 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:49.149182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33927 (local address 127.0.98.62:45839)
0925 12:17:49.149298 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:49.149299 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:49.149307 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:49.149350 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:49.149351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:49.149360 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:49.149414 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:49.149686 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:49.150408 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:49.151216 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:49.151398 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:49.151447 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:49.157247 (+ 5800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33927: BlockingRecv error: recv got EOF from 127.0.98.1:33927 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":64,"threads_started":1}
Sep 25 12:17:49 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:49 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:49.272589 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:49.268434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.2:48489)
0925 12:17:49.268544 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:49.268554 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:49.268612 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:49.268709 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:49.268710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:49.268848 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:49.268891 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:49.268893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:49.269457 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:49.269458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:49.269988 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:49.269992 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:49.270304 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:49.270304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:49.270340 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:49.270721 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:49.270725 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:49.272507 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
W20260925 12:17:49.272670 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:49.268479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48489 (local address 127.0.98.62:45839)
0925 12:17:49.268544 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:49.268546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:49.268613 (+ 67us) server_negotiation.cc:407] Connection header received
0925 12:17:49.268661 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:49.268661 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:49.268671 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:49.268705 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:49.268942 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:49.269396 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:49.270079 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:49.270246 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:49.270294 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:49.272574 (+ 2280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48489: BlockingRecv error: recv got EOF from 127.0.98.2:48489 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 12:17:49 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:49 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:49.393302 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:49.388965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.3:54247)
0925 12:17:49.389116 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:49.389128 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:49.389192 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:49.389323 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:49.389324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:49.389499 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:49.389541 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:49.389543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:49.390168 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:49.390170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:49.390687 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:49.390688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:49.391048 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:49.391049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:49.391105 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:49.391509 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:49.391513 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:49.393199 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":73,"threads_started":1}
W20260925 12:17:49.393339 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:49.389047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54247 (local address 127.0.98.62:45839)
0925 12:17:49.389118 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:49.389120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:49.389174 (+ 54us) server_negotiation.cc:407] Connection header received
0925 12:17:49.389244 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:49.389245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:49.389259 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:49.389312 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:49.389616 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:49.390089 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:49.390770 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:49.390965 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:49.391048 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:49.393279 (+ 2231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54247: BlockingRecv error: recv got EOF from 127.0.98.3:54247 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 12:17:50 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:50 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:50.161856 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:50.157667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.1:40319)
0925 12:17:50.157780 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:50.157791 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:50.157862 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:50.158002 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:50.158002 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:50.158161 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:50.158208 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:50.158209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:50.158805 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:50.158806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:50.159295 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:50.159296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:50.159604 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:50.159604 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:50.159699 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:50.160172 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:50.160176 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:50.161801 (+ 1625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
W20260925 12:17:50.161988 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:50.157735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40319 (local address 127.0.98.62:45839)
0925 12:17:50.157861 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:50.157863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:50.157871 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:50.157908 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:50.157909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:50.157919 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:50.157975 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:50.158271 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:50.158738 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:50.159372 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:50.159547 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:50.159594 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:50.161906 (+ 2312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40319: BlockingRecv error: recv got EOF from 127.0.98.1:40319 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
Sep 25 12:17:50 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:50 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:50.277529 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:50.273139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.2:55853)
0925 12:17:50.273258 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:50.273270 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:50.273336 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:50.273459 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:50.273460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:50.273636 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:50.273682 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:50.273683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:50.274394 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:50.274395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:50.274905 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:50.274907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:50.275220 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:50.275221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:50.275265 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:50.275819 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:50.275824 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:50.277475 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":65,"threads_started":1}
W20260925 12:17:50.277599 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:50.273157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55853 (local address 127.0.98.62:45839)
0925 12:17:50.273201 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:50.273204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:50.273321 (+ 117us) server_negotiation.cc:407] Connection header received
0925 12:17:50.273394 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:50.273394 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:50.273405 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:50.273444 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:50.273743 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:50.274323 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:50.274970 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:50.275158 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:50.275206 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:50.277556 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55853: BlockingRecv error: recv got EOF from 127.0.98.2:55853 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 12:17:50 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:50 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:50.397806 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:50.393715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.3:46531)
0925 12:17:50.393876 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:50.393887 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:50.393942 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:50.394067 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:50.394068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:50.394214 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:50.394262 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:50.394264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:50.394869 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:50.394870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:50.395381 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:50.395382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:50.395704 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:50.395704 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:50.395742 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:50.396144 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:50.396148 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:50.397720 (+ 1572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
W20260925 12:17:50.397850 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:50.393791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46531 (local address 127.0.98.62:45839)
0925 12:17:50.393863 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:50.393865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:50.393942 (+ 77us) server_negotiation.cc:407] Connection header received
0925 12:17:50.393989 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:50.393990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:50.394002 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:50.394048 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:50.394330 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:50.394807 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:50.395441 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:50.395628 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:50.395696 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:50.397790 (+ 2094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46531: BlockingRecv error: recv got EOF from 127.0.98.3:46531 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 12:17:51 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:51 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:51.166659 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:51.162468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.1:38771)
0925 12:17:51.162621 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:51.162631 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:51.162678 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:51.162838 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:51.162839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:51.163013 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:51.163057 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:51.163059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:51.163708 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:51.163709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:51.164275 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:51.164277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:51.164595 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:51.164596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:51.164634 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:51.165043 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:51.165047 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:51.166604 (+ 1557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
W20260925 12:17:51.166759 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:51.162577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38771 (local address 127.0.98.62:45839)
0925 12:17:51.162695 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:51.162697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:51.162724 (+ 27us) server_negotiation.cc:407] Connection header received
0925 12:17:51.162745 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:51.162746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:51.162757 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:51.162834 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:51.163116 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:51.163611 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:51.164349 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:51.164536 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:51.164584 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:51.166685 (+ 2101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38771: BlockingRecv error: recv got EOF from 127.0.98.1:38771 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
Sep 25 12:17:51 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:51 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:51.282573 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:51.278078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.2:57441)
0925 12:17:51.278202 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:51.278212 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:51.278254 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:51.278433 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:51.278433 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:51.278595 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:51.278638 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:51.278640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:51.279427 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:51.279429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:51.279922 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:51.279924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:51.280299 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:51.280300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:51.280386 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:51.280855 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:51.280860 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:51.282483 (+ 1623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
W20260925 12:17:51.282609 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:51.278248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57441 (local address 127.0.98.62:45839)
0925 12:17:51.278319 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:51.278320 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:51.278328 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:51.278354 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:51.278355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:51.278364 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:51.278425 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:51.278730 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:51.279338 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:51.280005 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:51.280206 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:51.280270 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:51.282552 (+ 2282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57441: BlockingRecv error: recv got EOF from 127.0.98.2:57441 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 12:17:51 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:51 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:51.406840 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:51.398222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.3:52231)
0925 12:17:51.398360 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:51.398371 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:51.398431 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:51.398579 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:51.398579 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:51.398741 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:51.398780 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:51.398782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:51.399400 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:51.399402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:51.400037 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:51.400039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:51.400407 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:51.400409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:51.400479 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:51.405201 (+ 4722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:51.405206 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:51.406791 (+ 1585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
W20260925 12:17:51.406965 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:51.398281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52231 (local address 127.0.98.62:45839)
0925 12:17:51.398354 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:51.398356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:51.398412 (+ 56us) server_negotiation.cc:407] Connection header received
0925 12:17:51.398493 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:51.398505 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:51.398517 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:51.398573 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:51.398872 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:51.399319 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:51.400115 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:51.400308 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:51.400408 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:51.406889 (+ 6481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52231: BlockingRecv error: recv got EOF from 127.0.98.3:52231 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 12:17:52 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:52 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:52.171263 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:52.167202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.1:35005)
0925 12:17:52.167344 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:52.167355 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:52.167398 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:52.167539 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:52.167540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:52.167723 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:52.167770 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:52.167772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:52.168348 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:52.168348 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:52.168818 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:52.168819 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:52.169113 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:52.169114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:52.169146 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:52.169514 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:52.169518 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:52.171168 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
W20260925 12:17:52.171306 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:52.167284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35005 (local address 127.0.98.62:45839)
0925 12:17:52.167411 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:52.167412 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:52.167436 (+ 24us) server_negotiation.cc:407] Connection header received
0925 12:17:52.167465 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:52.167466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:52.167475 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:52.167522 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:52.167828 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:52.168285 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:52.168894 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:52.169059 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:52.169101 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:52.171256 (+ 2155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35005: BlockingRecv error: recv got EOF from 127.0.98.1:35005 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":81,"threads_started":1}
Sep 25 12:17:52 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:52 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:52.287601 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:52.283289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.2:56821)
0925 12:17:52.283439 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:52.283452 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:52.283556 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:52.283764 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:52.283764 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:52.283925 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:52.283982 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:52.283984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:52.284607 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:52.284608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:52.285161 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:52.285163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:52.285458 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:52.285459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:52.285495 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:52.285895 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:52.285899 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:52.287520 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
W20260925 12:17:52.287743 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:52.283366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56821 (local address 127.0.98.62:45839)
0925 12:17:52.283428 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:52.283430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:52.283555 (+ 125us) server_negotiation.cc:407] Connection header received
0925 12:17:52.283677 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:52.283678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:52.283689 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:52.283742 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:52.284051 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:52.284526 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:52.285229 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:52.285399 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:52.285446 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:52.287677 (+ 2231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56821: BlockingRecv error: recv got EOF from 127.0.98.2:56821 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 12:17:52 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:52 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:52.411674 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:52.407299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.3:48075)
0925 12:17:52.407446 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:52.407456 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:52.407511 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:52.407680 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:52.407680 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:52.407825 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:52.407880 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:52.407881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:52.408495 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:52.408496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:52.408974 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:52.408976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:52.409249 (+ 273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:52.409250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:52.409284 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:52.409807 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:52.409811 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:52.411566 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
W20260925 12:17:52.411724 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:52.407372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48075 (local address 127.0.98.62:45839)
0925 12:17:52.407466 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:52.407468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:52.407499 (+ 31us) server_negotiation.cc:407] Connection header received
0925 12:17:52.407563 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:52.407564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:52.407575 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:52.407636 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:52.407961 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:52.408437 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:52.409037 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:52.409196 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:52.409239 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:52.411666 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48075: BlockingRecv error: recv got EOF from 127.0.98.3:48075 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 12:17:53 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:53 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:53.176529 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:53.171820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.1:60131)
0925 12:17:53.171980 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:53.171991 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:53.172060 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:53.172221 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:53.172222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:53.172428 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:53.172491 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:53.172493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:53.173140 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:53.173141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:53.173933 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:53.173935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:53.174266 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:53.174267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:53.174306 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:53.174796 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:53.174800 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:53.176474 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
W20260925 12:17:53.176656 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:53.171874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60131 (local address 127.0.98.62:45839)
0925 12:17:53.172011 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:53.172013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:53.172043 (+ 30us) server_negotiation.cc:407] Connection header received
0925 12:17:53.172120 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:53.172121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:53.172133 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:53.172206 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:53.172562 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:53.173085 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:53.174005 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:53.174205 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:53.174256 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:53.176552 (+ 2296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60131: BlockingRecv error: recv got EOF from 127.0.98.1:60131 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
Sep 25 12:17:53 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:53 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:53.292516 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:53.288077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.2:58157)
0925 12:17:53.288233 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:53.288242 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:53.288294 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:53.288424 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:53.288424 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:53.288575 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:53.288621 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:53.288623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:53.289226 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:53.289227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:53.289764 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:53.289765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:53.290144 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:53.290145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:53.290215 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:53.290707 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:53.290712 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:53.292459 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
W20260925 12:17:53.292645 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:53.288147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58157 (local address 127.0.98.62:45839)
0925 12:17:53.288207 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:53.288209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:53.288300 (+ 91us) server_negotiation.cc:407] Connection header received
0925 12:17:53.288353 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:53.288354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:53.288364 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:53.288405 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:53.288679 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:53.289154 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:53.289842 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:53.290055 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:53.290131 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:53.292565 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58157: BlockingRecv error: recv got EOF from 127.0.98.2:58157 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 12:17:53 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:53 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:53.416483 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:53.412284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.3:52215)
0925 12:17:53.412433 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:53.412451 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:53.412503 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:53.412624 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:53.412624 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:53.412780 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:53.412825 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:53.412827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:53.413431 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:53.413432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:53.413942 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:53.413944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:53.414262 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:53.414263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:53.414300 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:53.414709 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:53.414713 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:53.416430 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
W20260925 12:17:53.416599 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:53.412379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52215 (local address 127.0.98.62:45839)
0925 12:17:53.412449 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:53.412451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:53.412493 (+ 42us) server_negotiation.cc:407] Connection header received
0925 12:17:53.412545 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:53.412546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:53.412557 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:53.412607 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:53.412887 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:53.413364 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:53.414012 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:53.414201 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:53.414250 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:53.416512 (+ 2262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52215: BlockingRecv error: recv got EOF from 127.0.98.3:52215 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 12:17:54 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:54 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:54.184258 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:54.176910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.1:44061)
0925 12:17:54.177070 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:54.177083 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:54.177148 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:54.177264 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:54.177265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:54.177412 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:54.177459 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:54.177460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:54.178011 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:54.178012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:54.178478 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:54.178479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:54.178832 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:54.178833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:54.178913 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:54.182456 (+ 3543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:54.182460 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:54.184202 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
W20260925 12:17:54.184389 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:54.176982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44061 (local address 127.0.98.62:45839)
0925 12:17:54.177088 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:54.177090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:54.177148 (+ 58us) server_negotiation.cc:407] Connection header received
0925 12:17:54.177187 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:54.177188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:54.177197 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:54.177251 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:54.177512 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:54.177954 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:54.178531 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:54.178761 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:54.178826 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:54.184297 (+ 5471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44061: BlockingRecv error: recv got EOF from 127.0.98.1:44061 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":43,"threads_started":1}
Sep 25 12:17:54 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:54 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:54.297200 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:54.293043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.2:44683)
0925 12:17:54.293182 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:54.293192 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:54.293253 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:54.293385 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:54.293386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:54.293531 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:54.293576 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:54.293577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:54.294217 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:54.294217 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:54.294722 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:54.294724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:54.295027 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:54.295028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:54.295065 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:54.295497 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:54.295501 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:54.297149 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
W20260925 12:17:54.297293 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:54.293113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44683 (local address 127.0.98.62:45839)
0925 12:17:54.293175 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:54.293177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:54.293256 (+ 79us) server_negotiation.cc:407] Connection header received
0925 12:17:54.293297 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:54.293298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:54.293308 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:54.293363 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:54.293633 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:54.294150 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:54.294788 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:54.294969 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:54.295018 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:54.297232 (+ 2214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44683: BlockingRecv error: recv got EOF from 127.0.98.2:44683 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 25 12:17:54 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:54 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:54.421320 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:54.417057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.3:42617)
0925 12:17:54.417181 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:54.417191 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:54.417252 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:54.417371 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:54.417372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:54.417525 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:54.417568 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:54.417570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:54.418150 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:54.418151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:54.418870 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:54.418874 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:54.419162 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:54.419163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:54.419201 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:54.419581 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:54.419585 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:54.421236 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
W20260925 12:17:54.421362 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:54.417119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42617 (local address 127.0.98.62:45839)
0925 12:17:54.417180 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:54.417181 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:54.417253 (+ 72us) server_negotiation.cc:407] Connection header received
0925 12:17:54.417292 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:54.417293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:54.417305 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:54.417361 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:54.417622 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:54.418053 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:54.418937 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:54.419102 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:54.419150 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:54.421300 (+ 2150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42617: BlockingRecv error: recv got EOF from 127.0.98.3:42617 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 12:17:55 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:55 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:55.188900 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:55.184700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.1:40199)
0925 12:17:55.184842 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:55.184852 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:55.184903 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:55.185029 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:55.185029 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:55.185175 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:55.185221 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:55.185222 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:55.185832 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:55.185833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:55.186331 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:55.186332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:55.186642 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:55.186642 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:55.186680 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:55.187094 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:55.187098 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:55.188836 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
W20260925 12:17:55.189028 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:55.184783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40199 (local address 127.0.98.62:45839)
0925 12:17:55.184894 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:55.184896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:55.184904 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:55.184945 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:55.184946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:55.184956 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:55.185011 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:55.185281 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:55.185772 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:55.186403 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:55.186585 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:55.186632 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:55.188921 (+ 2289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40199: BlockingRecv error: recv got EOF from 127.0.98.1:40199 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":47,"threads_started":1}
Sep 25 12:17:55 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:55 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:55.302595 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:55.297704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.2:57375)
0925 12:17:55.297879 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:55.297893 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:55.297988 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:55.298157 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:55.298157 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:55.298338 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:55.298403 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:55.298404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:55.299250 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:55.299251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:55.299855 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:55.299858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:55.300220 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:55.300221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:55.300284 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:55.300733 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:55.300737 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:55.302541 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":93,"threads_started":1}
W20260925 12:17:55.302717 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:55.297806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57375 (local address 127.0.98.62:45839)
0925 12:17:55.297883 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:55.297886 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:55.297967 (+ 81us) server_negotiation.cc:407] Connection header received
0925 12:17:55.298069 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:55.298070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:55.298082 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:55.298151 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:55.298485 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:55.299177 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:55.299943 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:55.300139 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:55.300220 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:55.302638 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57375: BlockingRecv error: recv got EOF from 127.0.98.2:57375 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 12:17:55 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:55 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:55.426275 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:55.421839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.3:50929)
0925 12:17:55.422018 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:55.422029 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:55.422113 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:55.422262 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:55.422263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:55.422427 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:55.422470 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:55.422471 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:55.423104 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:55.423105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:55.423632 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:55.423633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:55.423997 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:55.423998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:55.424081 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:55.424570 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:55.424574 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:55.426224 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":97,"threads_started":1}
W20260925 12:17:55.426400 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:55.421933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50929 (local address 127.0.98.62:45839)
0925 12:17:55.422006 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:55.422009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:55.422096 (+ 87us) server_negotiation.cc:407] Connection header received
0925 12:17:55.422192 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:55.422193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:55.422207 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:55.422247 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:55.422544 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:55.423026 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:55.423715 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:55.423905 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:55.423997 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:55.426303 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50929: BlockingRecv error: recv got EOF from 127.0.98.3:50929 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 12:17:56 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:56 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:56.193907 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:56.189416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.1:46405)
0925 12:17:56.189565 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:56.189576 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:56.189639 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:56.189895 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:56.189896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:56.190102 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:56.190162 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:56.190164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:56.190819 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:56.190820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:56.191335 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:56.191336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:56.191686 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:56.191687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:56.191744 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:56.192164 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:56.192169 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:56.193853 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
W20260925 12:17:56.194048 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:56.189560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46405 (local address 127.0.98.62:45839)
0925 12:17:56.189694 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:56.189696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:56.189709 (+ 13us) server_negotiation.cc:407] Connection header received
0925 12:17:56.189747 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:56.189748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:56.189761 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:56.189884 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:56.190230 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:56.190752 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:56.191401 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:56.191582 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:56.191671 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:56.193958 (+ 2287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46405: BlockingRecv error: recv got EOF from 127.0.98.1:46405 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
Sep 25 12:17:56 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:56 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:56.307431 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:56.303228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.2:57551)
0925 12:17:56.303376 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:56.303386 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:56.303438 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:56.303546 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:56.303547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:56.303709 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:56.303753 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:56.303754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:56.304451 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:56.304452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:56.304974 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:56.304976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:56.305308 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:56.305309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:56.305355 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:56.305798 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:56.305802 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:56.307348 (+ 1546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
W20260925 12:17:56.307560 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:56.303340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57551 (local address 127.0.98.62:45839)
0925 12:17:56.303394 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:56.303396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:56.303425 (+ 29us) server_negotiation.cc:407] Connection header received
0925 12:17:56.303491 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:56.303492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:56.303501 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:56.303537 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:56.303808 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:56.304395 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:56.305030 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:56.305248 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:56.305297 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:56.307469 (+ 2172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57551: BlockingRecv error: recv got EOF from 127.0.98.2:57551 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 25 12:17:56 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:56 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:56.431190 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:56.426721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.3:43327)
0925 12:17:56.426850 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:56.426861 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:56.426927 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:56.427047 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:56.427048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:56.427194 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:56.427245 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:56.427247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:56.427852 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:56.427853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:56.428352 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:56.428354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:56.428659 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:56.428660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:56.428699 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:56.429195 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:56.429199 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:56.431119 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
W20260925 12:17:56.431267 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:56.426777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43327 (local address 127.0.98.62:45839)
0925 12:17:56.426845 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:56.426847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:56.426924 (+ 77us) server_negotiation.cc:407] Connection header received
0925 12:17:56.426969 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:56.426970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:56.426981 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:56.427031 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:56.427303 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:56.427789 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:56.428421 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:56.428599 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:56.428645 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:56.431214 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43327: BlockingRecv error: recv got EOF from 127.0.98.3:43327 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 12:17:57 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:57 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:57.198539 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:57.194385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.1:38699)
0925 12:17:57.194507 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:57.194519 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:57.194614 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:57.194761 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:57.194762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:57.194916 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:57.194962 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:57.194963 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:57.195588 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:57.195589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:57.196150 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:57.196152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:57.196475 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:57.196476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:57.196513 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:57.196910 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:57.196915 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:57.198487 (+ 1572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
W20260925 12:17:57.198634 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:57.194463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38699 (local address 127.0.98.62:45839)
0925 12:17:57.194590 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:57.194591 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:57.194600 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:17:57.194675 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:57.194676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:57.194686 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:57.194742 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:57.195026 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:57.195517 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:57.196231 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:57.196415 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:57.196464 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:57.198563 (+ 2099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38699: BlockingRecv error: recv got EOF from 127.0.98.1:38699 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
Sep 25 12:17:57 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:57 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:57.312752 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:57.307884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.2:59893)
0925 12:17:57.308033 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:57.308067 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:57.308129 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:57.308363 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:57.308364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:57.308546 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:57.308593 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:57.308595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:57.309314 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:57.309315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:57.309897 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:57.309900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:57.310334 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:57.310335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:57.310395 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:57.310916 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:57.310920 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:57.312658 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
W20260925 12:17:57.312809 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:57.308027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59893 (local address 127.0.98.62:45839)
0925 12:17:57.308102 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:57.308104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:57.308114 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:17:57.308238 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:57.308239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:57.308271 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:57.308332 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:57.308680 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:57.309185 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:57.310011 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:57.310233 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:57.310320 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:57.312737 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59893: BlockingRecv error: recv got EOF from 127.0.98.2:59893 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 12:17:57 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:57 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:57.436124 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:57.431724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.3:56303)
0925 12:17:57.431876 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:57.431888 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:57.431953 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:57.432091 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:57.432091 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:57.432238 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:57.432291 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:57.432293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:57.432924 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:57.432925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:57.433444 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:57.433445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:57.433778 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:57.433779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:57.433816 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:57.434265 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:57.434270 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:57.436020 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":93,"threads_started":1}
W20260925 12:17:57.436223 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:57.431793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56303 (local address 127.0.98.62:45839)
0925 12:17:57.431867 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:57.431868 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:57.431955 (+ 87us) server_negotiation.cc:407] Connection header received
0925 12:17:57.432009 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:57.432010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:57.432024 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:57.432068 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:57.432358 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:57.432844 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:57.433527 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:57.433705 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:57.433763 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:57.436146 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56303: BlockingRecv error: recv got EOF from 127.0.98.3:56303 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 12:17:58 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:58 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:58.203457 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:58.199154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.1:59581)
0925 12:17:58.199285 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:58.199294 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:58.199342 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:58.199497 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:58.199498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:58.199715 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:58.199762 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:58.199763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.200388 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.200389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:58.200862 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:58.200863 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.201233 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.201233 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:58.201312 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:58.201719 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:58.201722 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:58.203353 (+ 1631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
W20260925 12:17:58.203495 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:58.199258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59581 (local address 127.0.98.62:45839)
0925 12:17:58.199385 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:58.199387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:58.199395 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:17:58.199414 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:58.199415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:58.199425 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:58.199487 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:58.199839 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.200306 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.200940 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.201152 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.201234 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:58.203426 (+ 2192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59581: BlockingRecv error: recv got EOF from 127.0.98.1:59581 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":53,"threads_started":1}
Sep 25 12:17:58 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:17:58 dist-test-slave-clmx krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:17:58.317920 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:58.313336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45839 (local address 127.0.98.2:49887)
0925 12:17:58.313507 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:58.313520 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:58.313593 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:58.313735 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:58.313735 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:58.313900 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:58.313944 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:58.313946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.314617 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.314618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:58.315144 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:58.315146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.315524 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.315525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:58.315592 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:58.316106 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:17:58.316110 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:58.317830 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45839: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":85,"threads_started":1}
W20260925 12:17:58.317971 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:58.313429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49887 (local address 127.0.98.62:45839)
0925 12:17:58.313511 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:58.313514 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:58.313579 (+ 65us) server_negotiation.cc:407] Connection header received
0925 12:17:58.313646 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:58.313648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:58.313659 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:58.313725 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:58.314021 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.314507 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.315233 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.315437 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.315522 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:58.317911 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49887: BlockingRecv error: recv got EOF from 127.0.98.2:49887 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 12:17:58.360724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 9113
I20260925 12:17:58.364988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 9265
I20260925 12:17:58.368805 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 9398
I20260925 12:17:58.372685 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 9043
2026-09-25T12:17:58Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17307 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T12:17:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:58Z Disabled control of system clock
I20260925 12:17:58.398738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33997
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 12:17:58.473371 9593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:58.473552 9593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:58.473570 9593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:58.474639 9593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:58.474673 9593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:58.474684 9593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:58.474696 9593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:58.474709 9593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:17:58.475956 9593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33997
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:58.476117 9593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:58.476322 9593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:58.481082 9598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:58.481089 9599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:58.481299 9593 server_base.cc:1034] running on GCE node
W20260925 12:17:58.481491 9601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:58.481732 9593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:58.482048 9593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:58.483206 9593 hybrid_clock.cc:648] HybridClock initialized: now 1790338678483188 us; error 29 us; skew 500 ppm
I20260925 12:17:58.484376 9593 webserver.cc:460] Webserver started at http://127.0.98.62:42569/ using document root <none> and password file <none>
I20260925 12:17:58.484612 9593 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:58.484678 9593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:58.484822 9593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:58.485791 9593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "467f2f92d1c0494894e936041fed3434"
format_stamp: "Formatted at 2026-09-25 12:17:58 on dist-test-slave-clmx"
server_key: "42dd0e86970c9cbcfa3f4cb096094a62"
server_key_iv: "2ab40979742a963bab8b5fa7797d0a3d"
server_key_version: "encryptionkey@0"
I20260925 12:17:58.486132 9593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "467f2f92d1c0494894e936041fed3434"
format_stamp: "Formatted at 2026-09-25 12:17:58 on dist-test-slave-clmx"
server_key: "42dd0e86970c9cbcfa3f4cb096094a62"
server_key_iv: "2ab40979742a963bab8b5fa7797d0a3d"
server_key_version: "encryptionkey@0"
I20260925 12:17:58.487349 9593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:17:58.488106 9607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:58.488262 9593 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:58.488360 9593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "467f2f92d1c0494894e936041fed3434"
format_stamp: "Formatted at 2026-09-25 12:17:58 on dist-test-slave-clmx"
server_key: "42dd0e86970c9cbcfa3f4cb096094a62"
server_key_iv: "2ab40979742a963bab8b5fa7797d0a3d"
server_key_version: "encryptionkey@0"
I20260925 12:17:58.488441 9593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:58.499197 9593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:58.499428 9593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:58.499555 9593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:58.503185 9593 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33997
I20260925 12:17:58.503216 9659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33997 every 8 connection(s)
I20260925 12:17:58.503522 9593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:58.504206 9660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:58.506477 9660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434: Bootstrap starting.
I20260925 12:17:58.506953 9660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:58.507166 9660 log.cc:824] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:58.507786 9660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434: No bootstrap required, opened a new log
I20260925 12:17:58.509163 9660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467f2f92d1c0494894e936041fed3434" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } }
I20260925 12:17:58.509276 9660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:58.509296 9660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 467f2f92d1c0494894e936041fed3434, State: Initialized, Role: FOLLOWER
I20260925 12:17:58.509414 9660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467f2f92d1c0494894e936041fed3434" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } }
I20260925 12:17:58.509497 9660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:58.509518 9660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:58.509543 9660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:58.509984 9660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467f2f92d1c0494894e936041fed3434" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } }
I20260925 12:17:58.510058 9660 leader_election.cc:302] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 467f2f92d1c0494894e936041fed3434; no voters:
I20260925 12:17:58.510161 9660 leader_election.cc:288] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:17:58.510293 9663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:58.510440 9663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434 [term 1 LEADER]: Becoming Leader. State: Replica: 467f2f92d1c0494894e936041fed3434, State: Running, Role: LEADER
I20260925 12:17:58.510464 9660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:58.510525 9663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467f2f92d1c0494894e936041fed3434" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } }
I20260925 12:17:58.510892 9664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "467f2f92d1c0494894e936041fed3434" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467f2f92d1c0494894e936041fed3434" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } }
I20260925 12:17:58.511036 9665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 467f2f92d1c0494894e936041fed3434. Latest consensus state: current_term: 1 leader_uuid: "467f2f92d1c0494894e936041fed3434" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467f2f92d1c0494894e936041fed3434" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } }
I20260925 12:17:58.511147 9665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:58.511045 9664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:58.511516 9672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:58.511927 9672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:17:58.512316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 9593
I20260925 12:17:58.512410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:17:58.512598 392 external_mini_cluster.cc:1428] Setting key 68f724acbd26b696d015669abc236048
I20260925 12:17:58.513648 9672 catalog_manager.cc:1347] Generated new cluster ID: a0edea7c6b7642ae95fc2c3c7486c95d
I20260925 12:17:58.513695 9672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:17:58.524755 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:58.512974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60206 (local address 127.0.98.62:33997)
0925 12:17:58.519766 (+ 6792us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:58.519775 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:58.519806 (+ 31us) server_negotiation.cc:407] Connection header received
0925 12:17:58.519917 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:58.519923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:58.519985 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:58.520117 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:58.520342 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.521689 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.522694 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.523305 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.523419 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:58.524063 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:58.524067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:58.524067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:58.524267 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:58.524321 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:58.524335 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:58.524403 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:58.524468 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 12:17:58.524536 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6713,"thread_start_us":63,"threads_started":1}
I20260925 12:17:58.529734 9672 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:17:58.530329 9672 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:58.532222 9672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 467f2f92d1c0494894e936041fed3434: Generated new TSK 0
I20260925 12:17:58.532378 9672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:17:58.535395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33997
--builtin_ntp_servers=127.0.98.20:44755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 12:17:58.611877 9684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:58.612063 9684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:58.612082 9684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:58.613150 9684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:58.613184 9684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:58.613207 9684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:17:58.614428 9684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:58.614653 9684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:58.614938 9684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:58.619362 9689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:58.619344 9690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:58.619665 9684 server_base.cc:1034] running on GCE node
W20260925 12:17:58.619699 9692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:58.619913 9684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:58.620138 9684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:58.621266 9684 hybrid_clock.cc:648] HybridClock initialized: now 1790338678621246 us; error 37 us; skew 500 ppm
I20260925 12:17:58.622444 9684 webserver.cc:460] Webserver started at http://127.0.98.1:36747/ using document root <none> and password file <none>
I20260925 12:17:58.622684 9684 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:58.622745 9684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:58.622864 9684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:58.623884 9684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "f8142a2a38324345a7721e4510d04e4a"
format_stamp: "Formatted at 2026-09-25 12:17:58 on dist-test-slave-clmx"
server_key: "09522ddda061db493497d549388385ec"
server_key_iv: "0dc9ff0b395fced7f23d10785f676ec9"
server_key_version: "encryptionkey@0"
I20260925 12:17:58.624241 9684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f8142a2a38324345a7721e4510d04e4a"
format_stamp: "Formatted at 2026-09-25 12:17:58 on dist-test-slave-clmx"
server_key: "09522ddda061db493497d549388385ec"
server_key_iv: "0dc9ff0b395fced7f23d10785f676ec9"
server_key_version: "encryptionkey@0"
I20260925 12:17:58.625496 9684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:17:58.626235 9698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:58.626479 9684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:17:58.626569 9684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "f8142a2a38324345a7721e4510d04e4a"
format_stamp: "Formatted at 2026-09-25 12:17:58 on dist-test-slave-clmx"
server_key: "09522ddda061db493497d549388385ec"
server_key_iv: "0dc9ff0b395fced7f23d10785f676ec9"
server_key_version: "encryptionkey@0"
I20260925 12:17:58.626637 9684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:58.638651 9684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:58.638901 9684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:58.639034 9684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:58.639232 9684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:58.639528 9684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:58.639580 9684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:58.639645 9684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:58.639669 9684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:58.644858 9684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37011
I20260925 12:17:58.644894 9811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37011 every 8 connection(s)
I20260925 12:17:58.645170 9684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:17:58.649868 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:58.645814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43761 (local address 127.0.98.62:33997)
0925 12:17:58.645875 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:58.645878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:58.646056 (+ 178us) server_negotiation.cc:407] Connection header received
0925 12:17:58.646499 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:58.646501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:58.646513 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:58.646572 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:58.647144 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.647676 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.648616 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.648789 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.648856 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:58.649586 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:58.649588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:58.649588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:58.649592 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:58.649616 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:58.649620 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:58.649676 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:58.649771 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 12:17:58.649800 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:17:58.649870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 9684
I20260925 12:17:58.649974 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:17:58.650086 9814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:58.645785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33997 (local address 127.0.98.1:43761)
0925 12:17:58.645947 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:58.645981 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:58.646406 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:58.646630 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:58.646633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:58.646642 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:58.647057 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:58.647061 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.647775 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.647776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:58.648522 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:58.648524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.648869 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.648870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:58.648990 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:58.649463 (+ 473us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:17:58.649463 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:58.649486 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:17:58.649505 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:58.649677 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:58.649682 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:58.649685 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:58.649760 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 12:17:58.649847 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260925 12:17:58.650168 392 external_mini_cluster.cc:1428] Setting key 237807f78a4bf1631ebdff6312a9afc6
I20260925 12:17:58.650512 9812 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33997
I20260925 12:17:58.650606 9812 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:58.650770 9812 heartbeater.cc:511] Master 127.0.98.62:33997 requested a full tablet report, sending...
I20260925 12:17:58.651049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33997
--builtin_ntp_servers=127.0.98.20:44755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 12:17:58.651297 9624 ts_manager.cc:194] Registered new tserver with Master: f8142a2a38324345a7721e4510d04e4a (127.0.98.1:37011)
I20260925 12:17:58.651888 9624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:43761
W20260925 12:17:58.726847 9815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:58.727022 9815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:58.727042 9815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:58.728118 9815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:58.728152 9815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:58.728173 9815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:17:58.729367 9815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:58.729557 9815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:58.729784 9815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:58.734036 9820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:58.734061 9821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:58.734068 9823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:58.734181 9815 server_base.cc:1034] running on GCE node
I20260925 12:17:58.734449 9815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:58.734700 9815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:58.735868 9815 hybrid_clock.cc:648] HybridClock initialized: now 1790338678735834 us; error 59 us; skew 500 ppm
I20260925 12:17:58.736912 9815 webserver.cc:460] Webserver started at http://127.0.98.2:35315/ using document root <none> and password file <none>
I20260925 12:17:58.737138 9815 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:58.737213 9815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:58.737340 9815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:58.738260 9815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "9e9a49fe13e04632830c18fa4883409f"
format_stamp: "Formatted at 2026-09-25 12:17:58 on dist-test-slave-clmx"
server_key: "38d3c7e878c0dc0c64c630987fc37a18"
server_key_iv: "51feb48cb98b8768f6d7dbc8e59aa296"
server_key_version: "encryptionkey@0"
I20260925 12:17:58.738587 9815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9e9a49fe13e04632830c18fa4883409f"
format_stamp: "Formatted at 2026-09-25 12:17:58 on dist-test-slave-clmx"
server_key: "38d3c7e878c0dc0c64c630987fc37a18"
server_key_iv: "51feb48cb98b8768f6d7dbc8e59aa296"
server_key_version: "encryptionkey@0"
I20260925 12:17:58.739735 9815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 12:17:58.740350 9829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:58.740509 9815 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:58.740592 9815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "9e9a49fe13e04632830c18fa4883409f"
format_stamp: "Formatted at 2026-09-25 12:17:58 on dist-test-slave-clmx"
server_key: "38d3c7e878c0dc0c64c630987fc37a18"
server_key_iv: "51feb48cb98b8768f6d7dbc8e59aa296"
server_key_version: "encryptionkey@0"
I20260925 12:17:58.740664 9815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:58.747833 9815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:58.748052 9815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:58.748193 9815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:58.748379 9815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:58.748656 9815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:58.748704 9815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:58.748739 9815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:58.748764 9815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:58.754173 9815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41287
I20260925 12:17:58.754220 9942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41287 every 8 connection(s)
I20260925 12:17:58.754469 9815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:17:58.755270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 9815
I20260925 12:17:58.755362 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:17:58.755548 392 external_mini_cluster.cc:1428] Setting key 12f9edc252eaf6264eec1ab255e95032
I20260925 12:17:58.756517 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33997
--builtin_ntp_servers=127.0.98.20:44755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 12:17:58.759275 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:58.755087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42931 (local address 127.0.98.62:33997)
0925 12:17:58.755153 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:58.755155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:58.755360 (+ 205us) server_negotiation.cc:407] Connection header received
0925 12:17:58.755887 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:58.755888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:58.755918 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:58.755993 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:58.756608 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.757080 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.757971 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.758216 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.758288 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:58.759012 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:58.759013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:58.759014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:58.759018 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:58.759032 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:58.759036 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:58.759083 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:58.759175 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 12:17:58.759202 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 12:17:58.759467 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:58.755061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33997 (local address 127.0.98.2:42931)
0925 12:17:58.755238 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:58.755272 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:58.755780 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:58.756036 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:58.756039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:58.756046 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:58.756522 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:58.756526 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.757168 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.757169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:58.757883 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:58.757885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.758304 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.758305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:58.758428 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:58.758882 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:17:58.758882 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:58.758905 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:17:58.758925 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:58.759082 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:58.759086 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:58.759089 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:58.759175 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 12:17:58.759262 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260925 12:17:58.759879 9943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33997
I20260925 12:17:58.759969 9943 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:58.760142 9943 heartbeater.cc:511] Master 127.0.98.62:33997 requested a full tablet report, sending...
I20260925 12:17:58.760469 9624 ts_manager.cc:194] Registered new tserver with Master: 9e9a49fe13e04632830c18fa4883409f (127.0.98.2:41287)
I20260925 12:17:58.760969 9624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:42931
W20260925 12:17:58.833307 9946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:58.833493 9946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:58.833513 9946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:58.834537 9946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:58.834568 9946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:58.834592 9946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:17:58.835865 9946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:58.836018 9946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:58.836215 9946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:58.840689 9951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:58.840785 9952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:58.840690 9954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:58.840842 9946 server_base.cc:1034] running on GCE node
I20260925 12:17:58.841039 9946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:58.841212 9946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:58.842370 9946 hybrid_clock.cc:648] HybridClock initialized: now 1790338678842354 us; error 25 us; skew 500 ppm
I20260925 12:17:58.843442 9946 webserver.cc:460] Webserver started at http://127.0.98.3:41785/ using document root <none> and password file <none>
I20260925 12:17:58.843700 9946 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:58.843770 9946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:58.843912 9946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:58.844871 9946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "3c68c8009d264631b16329d0f1241c0d"
format_stamp: "Formatted at 2026-09-25 12:17:58 on dist-test-slave-clmx"
server_key: "a6743b0e0cea36ef07a45d7405981867"
server_key_iv: "69b02e8475c291b518ca94d16c15bfd8"
server_key_version: "encryptionkey@0"
I20260925 12:17:58.845203 9946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3c68c8009d264631b16329d0f1241c0d"
format_stamp: "Formatted at 2026-09-25 12:17:58 on dist-test-slave-clmx"
server_key: "a6743b0e0cea36ef07a45d7405981867"
server_key_iv: "69b02e8475c291b518ca94d16c15bfd8"
server_key_version: "encryptionkey@0"
I20260925 12:17:58.846374 9946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:17:58.846969 9960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:58.847110 9946 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:58.847191 9946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "3c68c8009d264631b16329d0f1241c0d"
format_stamp: "Formatted at 2026-09-25 12:17:58 on dist-test-slave-clmx"
server_key: "a6743b0e0cea36ef07a45d7405981867"
server_key_iv: "69b02e8475c291b518ca94d16c15bfd8"
server_key_version: "encryptionkey@0"
I20260925 12:17:58.847261 9946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:58.859743 9946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:58.859970 9946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:58.860096 9946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:58.860281 9946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:58.860548 9946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:58.860600 9946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:58.860632 9946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:58.860656 9946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:58.866081 9946 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38535
I20260925 12:17:58.866156 10073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38535 every 8 connection(s)
I20260925 12:17:58.866425 9946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:17:58.870539 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:58.867038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59345 (local address 127.0.98.62:33997)
0925 12:17:58.867080 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:58.867082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:58.867262 (+ 180us) server_negotiation.cc:407] Connection header received
0925 12:17:58.867656 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:58.867657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:58.867667 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:58.867731 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:58.868233 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.868670 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.869464 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.869646 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.869692 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:58.870322 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:58.870324 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:58.870324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:58.870328 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:58.870340 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:58.870343 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:58.870384 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:58.870459 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 12:17:58.870486 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:17:58.870693 10076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:58.866993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33997 (local address 127.0.98.3:59345)
0925 12:17:58.867149 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:58.867182 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:58.867553 (+ 371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:58.867776 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:58.867779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:58.867786 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:58.868166 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:58.868170 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.868735 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.868736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:58.869394 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:17:58.869396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:17:58.869709 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:17:58.869709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:17:58.869800 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:17:58.870204 (+ 404us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:17:58.870204 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:58.870227 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:17:58.870245 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:58.870393 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:58.870397 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:58.870399 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:17:58.870445 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 12:17:58.870509 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260925 12:17:58.871086 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 9946
I20260925 12:17:58.871182 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:17:58.871174 10074 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33997
I20260925 12:17:58.871251 10074 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:58.871378 392 external_mini_cluster.cc:1428] Setting key 8c5e112426c01cc52d8e775e2fb2324d
I20260925 12:17:58.871433 10074 heartbeater.cc:511] Master 127.0.98.62:33997 requested a full tablet report, sending...
I20260925 12:17:58.871791 9623 ts_manager.cc:194] Registered new tserver with Master: 3c68c8009d264631b16329d0f1241c0d (127.0.98.3:38535)
I20260925 12:17:58.872207 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:17:58.872243 9623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:59345
W20260925 12:17:58.873994 9682 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:58.873561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.0.98.62:33997)
0925 12:17:58.873614 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:58.873617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:58.873741 (+ 124us) server_negotiation.cc:407] Connection header received
0925 12:17:58.873772 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:58.873773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:58.873783 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:58.873821 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:58.873945 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60220: BlockingRecv error: recv got EOF from 127.0.0.1:60220 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 12:17:58.874045 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 9684
I20260925 12:17:58.878855 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 9815
I20260925 12:17:58.883093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 9946
I20260925 12:17:58.887023 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 9593
2026-09-25T12:17:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (514 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T12:17:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:17:58Z Disabled control of system clock
I20260925 12:17:58.912976 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41951
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 12:17:58.988664 10085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:58.988837 10085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:58.988855 10085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:58.989935 10085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:17:58.989967 10085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:58.989980 10085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:58.989992 10085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:17:58.990005 10085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:17:58.991205 10085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41951
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:58.991372 10085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:58.991575 10085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:58.995894 10091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:58.995922 10085 server_base.cc:1034] running on GCE node
W20260925 12:17:58.995891 10093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:58.995926 10090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:58.996309 10085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:58.996606 10085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:58.997761 10085 hybrid_clock.cc:648] HybridClock initialized: now 1790338678997740 us; error 33 us; skew 500 ppm
I20260925 12:17:58.998775 10085 webserver.cc:460] Webserver started at http://127.0.98.62:42255/ using document root <none> and password file <none>
I20260925 12:17:58.998984 10085 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:58.999046 10085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:58.999168 10085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:59.000108 10085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "62b5855be4ef4cc18c7f5cd1c4fc3917"
format_stamp: "Formatted at 2026-09-25 12:17:58 on dist-test-slave-clmx"
server_key: "19db8efe6562e74414734f065e0cc46b"
server_key_iv: "0a80c1d36e9f48cb796911bb3880b8b3"
server_key_version: "encryptionkey@0"
I20260925 12:17:59.000440 10085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "62b5855be4ef4cc18c7f5cd1c4fc3917"
format_stamp: "Formatted at 2026-09-25 12:17:58 on dist-test-slave-clmx"
server_key: "19db8efe6562e74414734f065e0cc46b"
server_key_iv: "0a80c1d36e9f48cb796911bb3880b8b3"
server_key_version: "encryptionkey@0"
I20260925 12:17:59.001500 10085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:17:59.002125 10099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:59.002290 10085 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:59.002388 10085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "62b5855be4ef4cc18c7f5cd1c4fc3917"
format_stamp: "Formatted at 2026-09-25 12:17:58 on dist-test-slave-clmx"
server_key: "19db8efe6562e74414734f065e0cc46b"
server_key_iv: "0a80c1d36e9f48cb796911bb3880b8b3"
server_key_version: "encryptionkey@0"
I20260925 12:17:59.002468 10085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:59.005630 10085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:59.005879 10085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:59.006007 10085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:59.009402 10085 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41951
I20260925 12:17:59.009443 10151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41951 every 8 connection(s)
I20260925 12:17:59.009711 10085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:17:59.010237 10152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:17:59.012579 10152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917: Bootstrap starting.
I20260925 12:17:59.013085 10152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917: Neither blocks nor log segments found. Creating new log.
I20260925 12:17:59.013350 10152 log.cc:824] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917: Log is configured to *not* fsync() on all Append() calls
I20260925 12:17:59.013931 10152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917: No bootstrap required, opened a new log
I20260925 12:17:59.015385 10152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b5855be4ef4cc18c7f5cd1c4fc3917" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41951 } }
I20260925 12:17:59.015517 10152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:17:59.015563 10152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62b5855be4ef4cc18c7f5cd1c4fc3917, State: Initialized, Role: FOLLOWER
I20260925 12:17:59.015726 10152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b5855be4ef4cc18c7f5cd1c4fc3917" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41951 } }
I20260925 12:17:59.015806 10152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:17:59.015843 10152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:17:59.015897 10152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:17:59.016381 10152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b5855be4ef4cc18c7f5cd1c4fc3917" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41951 } }
I20260925 12:17:59.016491 10152 leader_election.cc:302] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62b5855be4ef4cc18c7f5cd1c4fc3917; no voters:
I20260925 12:17:59.016521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 10085
I20260925 12:17:59.016604 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:17:59.016644 10152 leader_election.cc:288] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:17:59.016683 10155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:17:59.016808 392 external_mini_cluster.cc:1428] Setting key 33f1a4d44f48cd6e3e59652c7426ee41
I20260925 12:17:59.016820 10155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917 [term 1 LEADER]: Becoming Leader. State: Replica: 62b5855be4ef4cc18c7f5cd1c4fc3917, State: Running, Role: LEADER
I20260925 12:17:59.016872 10152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:17:59.016918 10155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b5855be4ef4cc18c7f5cd1c4fc3917" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41951 } }
I20260925 12:17:59.017198 10157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62b5855be4ef4cc18c7f5cd1c4fc3917. Latest consensus state: current_term: 1 leader_uuid: "62b5855be4ef4cc18c7f5cd1c4fc3917" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b5855be4ef4cc18c7f5cd1c4fc3917" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41951 } } }
I20260925 12:17:59.017367 10157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:59.017369 10156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62b5855be4ef4cc18c7f5cd1c4fc3917" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b5855be4ef4cc18c7f5cd1c4fc3917" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41951 } } }
I20260925 12:17:59.017498 10156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917 [sys.catalog]: This master's current role is: LEADER
I20260925 12:17:59.017719 10166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:17:59.018182 10166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:17:59.019526 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:59.017361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44526 (local address 127.0.98.62:41951)
0925 12:17:59.017569 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:59.017577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:59.017725 (+ 148us) server_negotiation.cc:407] Connection header received
0925 12:17:59.017942 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:59.017950 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:59.017997 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:59.018099 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:59.019086 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:59.019089 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:59.019089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:59.019093 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:59.019128 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:59.019140 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:59.019196 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:59.019322 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 12:17:59.019378 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260925 12:17:59.019790 10166 catalog_manager.cc:1347] Generated new cluster ID: 009bf44b9e7e4c4eaa767449c08b202a
I20260925 12:17:59.019838 10166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:17:59.039644 10166 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:17:59.039769 10166 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:17:59.043694 10166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62b5855be4ef4cc18c7f5cd1c4fc3917: Generated new TSK 0
I20260925 12:17:59.043848 10166 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:17:59.046818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41951
--builtin_ntp_servers=127.0.98.20:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 12:17:59.135028 10176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:59.135208 10176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:59.135229 10176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:59.136348 10176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:59.136380 10176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:59.136404 10176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:17:59.137567 10176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:59.137761 10176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:59.138028 10176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:17:59.142421 10181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:59.142421 10184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:59.142412 10182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:59.142786 10176 server_base.cc:1034] running on GCE node
I20260925 12:17:59.143036 10176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:59.143311 10176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:59.144466 10176 hybrid_clock.cc:648] HybridClock initialized: now 1790338679144455 us; error 30 us; skew 500 ppm
I20260925 12:17:59.145728 10176 webserver.cc:460] Webserver started at http://127.0.98.1:44589/ using document root <none> and password file <none>
I20260925 12:17:59.145972 10176 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:59.146014 10176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:59.146144 10176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:59.147104 10176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "bd2e300983bc4cd2ad980fa865904cb5"
format_stamp: "Formatted at 2026-09-25 12:17:59 on dist-test-slave-clmx"
server_key: "20667577f665d68c39667f5d22c2ccf6"
server_key_iv: "0c498743d7be7667a6eb8ef47c649c66"
server_key_version: "encryptionkey@0"
I20260925 12:17:59.147461 10176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bd2e300983bc4cd2ad980fa865904cb5"
format_stamp: "Formatted at 2026-09-25 12:17:59 on dist-test-slave-clmx"
server_key: "20667577f665d68c39667f5d22c2ccf6"
server_key_iv: "0c498743d7be7667a6eb8ef47c649c66"
server_key_version: "encryptionkey@0"
I20260925 12:17:59.148806 10176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:17:59.149550 10190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:59.149716 10176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:17:59.149809 10176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "bd2e300983bc4cd2ad980fa865904cb5"
format_stamp: "Formatted at 2026-09-25 12:17:59 on dist-test-slave-clmx"
server_key: "20667577f665d68c39667f5d22c2ccf6"
server_key_iv: "0c498743d7be7667a6eb8ef47c649c66"
server_key_version: "encryptionkey@0"
I20260925 12:17:59.149869 10176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:59.154486 10176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:59.154748 10176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:59.154870 10176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:59.155048 10176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:59.155315 10176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:59.155360 10176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:59.155386 10176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:59.155417 10176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:59.161773 10176 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40985
I20260925 12:17:59.161844 10303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40985 every 8 connection(s)
I20260925 12:17:59.162103 10176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:17:59.164988 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:59.162724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47309 (local address 127.0.98.62:41951)
0925 12:17:59.162786 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:59.162788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:59.163085 (+ 297us) server_negotiation.cc:407] Connection header received
0925 12:17:59.163540 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:59.163541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:59.163548 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:59.163608 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:59.164721 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:59.164723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:59.164723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:59.164729 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:59.164746 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:59.164751 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:59.164817 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:59.164901 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 12:17:59.164918 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 12:17:59.165163 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:59.162709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41951 (local address 127.0.98.1:47309)
0925 12:17:59.162964 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:59.162997 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:59.163438 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:59.163687 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:59.163691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:59.163700 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:59.164570 (+ 870us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:17:59.164571 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:59.164595 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:17:59.164620 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:59.164816 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:59.164820 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:59.164823 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:17:59.164906 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 12:17:59.164962 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":172,"threads_started":1}
I20260925 12:17:59.165788 10304 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41951
I20260925 12:17:59.165889 10304 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:59.166038 10304 heartbeater.cc:511] Master 127.0.98.62:41951 requested a full tablet report, sending...
I20260925 12:17:59.166543 10116 ts_manager.cc:194] Registered new tserver with Master: bd2e300983bc4cd2ad980fa865904cb5 (127.0.98.1:40985)
I20260925 12:17:59.171460 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 10176
I20260925 12:17:59.171679 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:17:59.171885 392 external_mini_cluster.cc:1428] Setting key 0a4c5f5ddc4ffca6134c557708e8e6dc
I20260925 12:17:59.172776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41951
--builtin_ntp_servers=127.0.98.20:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 12:17:59.246865 10307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:59.247028 10307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:59.247048 10307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:59.248054 10307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:59.248083 10307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:59.248106 10307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:17:59.249307 10307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:59.249471 10307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:59.249711 10307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:17:59.254266 10307 server_base.cc:1034] running on GCE node
W20260925 12:17:59.254256 10312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:59.254345 10313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:59.254243 10315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:59.254698 10307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:59.254927 10307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:59.256088 10307 hybrid_clock.cc:648] HybridClock initialized: now 1790338679256086 us; error 36 us; skew 500 ppm
I20260925 12:17:59.264708 10307 webserver.cc:460] Webserver started at http://127.0.98.2:36451/ using document root <none> and password file <none>
I20260925 12:17:59.264948 10307 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:59.265015 10307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:59.265162 10307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:59.266100 10307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "6424ced095d04a1d9ab54d67663c72fd"
format_stamp: "Formatted at 2026-09-25 12:17:59 on dist-test-slave-clmx"
server_key: "4e5209f7f3d7de1fda121d004a870469"
server_key_iv: "b234581121a2f935fa17789ac04fa759"
server_key_version: "encryptionkey@0"
I20260925 12:17:59.266453 10307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6424ced095d04a1d9ab54d67663c72fd"
format_stamp: "Formatted at 2026-09-25 12:17:59 on dist-test-slave-clmx"
server_key: "4e5209f7f3d7de1fda121d004a870469"
server_key_iv: "b234581121a2f935fa17789ac04fa759"
server_key_version: "encryptionkey@0"
I20260925 12:17:59.267603 10307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 12:17:59.268355 10321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:59.268512 10307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:17:59.268607 10307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "6424ced095d04a1d9ab54d67663c72fd"
format_stamp: "Formatted at 2026-09-25 12:17:59 on dist-test-slave-clmx"
server_key: "4e5209f7f3d7de1fda121d004a870469"
server_key_iv: "b234581121a2f935fa17789ac04fa759"
server_key_version: "encryptionkey@0"
I20260925 12:17:59.268685 10307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:59.271589 10307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:59.271876 10307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:59.272006 10307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:59.272204 10307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:59.272469 10307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:59.272522 10307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:59.272558 10307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:59.272584 10307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:59.277969 10307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36167
I20260925 12:17:59.278009 10434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36167 every 8 connection(s)
I20260925 12:17:59.278306 10307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:17:59.280783 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:59.278975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38639 (local address 127.0.98.62:41951)
0925 12:17:59.279015 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:59.279017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:59.279209 (+ 192us) server_negotiation.cc:407] Connection header received
0925 12:17:59.279572 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:59.279573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:59.279578 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:59.279680 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:59.280611 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:59.280612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:59.280613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:59.280616 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:59.280629 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:59.280632 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:59.280681 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:59.280725 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 12:17:59.280736 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 12:17:59.280969 10437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:59.278958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41951 (local address 127.0.98.2:38639)
0925 12:17:59.279102 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:59.279131 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:59.279499 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:59.279738 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:59.279741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:59.279748 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:59.280489 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:17:59.280489 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:59.280513 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:17:59.280536 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:59.280681 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:59.280683 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:59.280685 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:17:59.280731 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 12:17:59.280771 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":74,"threads_started":1}
I20260925 12:17:59.281431 10435 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41951
I20260925 12:17:59.281492 10435 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:59.281625 10435 heartbeater.cc:511] Master 127.0.98.62:41951 requested a full tablet report, sending...
I20260925 12:17:59.281940 10116 ts_manager.cc:194] Registered new tserver with Master: 6424ced095d04a1d9ab54d67663c72fd (127.0.98.2:36167)
I20260925 12:17:59.286789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 10307
I20260925 12:17:59.286867 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:17:59.287057 392 external_mini_cluster.cc:1428] Setting key 647823ddd9fdf435f038372a60ad2e43
I20260925 12:17:59.287935 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41951
--builtin_ntp_servers=127.0.98.20:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 12:17:59.363741 10438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:17:59.363895 10438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:17:59.363914 10438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:17:59.365077 10438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:17:59.365113 10438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:17:59.365163 10438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:17:59.366434 10438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:17:59.366652 10438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:17:59.366894 10438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:17:59.370956 10438 server_base.cc:1034] running on GCE node
W20260925 12:17:59.370940 10446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:59.371042 10444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:17:59.370922 10443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:17:59.371316 10438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:17:59.371588 10438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:17:59.372795 10438 hybrid_clock.cc:648] HybridClock initialized: now 1790338679372781 us; error 32 us; skew 500 ppm
I20260925 12:17:59.373860 10438 webserver.cc:460] Webserver started at http://127.0.98.3:33707/ using document root <none> and password file <none>
I20260925 12:17:59.374078 10438 fs_manager.cc:359] Metadata directory not provided
I20260925 12:17:59.374141 10438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:17:59.374266 10438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:17:59.375214 10438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "0bfcc5bb2e88449a8c8665afc312f505"
format_stamp: "Formatted at 2026-09-25 12:17:59 on dist-test-slave-clmx"
server_key: "6bc940d93358af3686cd4fedc6a8ae1e"
server_key_iv: "bedd639b95322d5a3cb62055e401bf9a"
server_key_version: "encryptionkey@0"
I20260925 12:17:59.375553 10438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0bfcc5bb2e88449a8c8665afc312f505"
format_stamp: "Formatted at 2026-09-25 12:17:59 on dist-test-slave-clmx"
server_key: "6bc940d93358af3686cd4fedc6a8ae1e"
server_key_iv: "bedd639b95322d5a3cb62055e401bf9a"
server_key_version: "encryptionkey@0"
I20260925 12:17:59.376784 10438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:17:59.377475 10452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:59.377635 10438 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:59.377722 10438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "0bfcc5bb2e88449a8c8665afc312f505"
format_stamp: "Formatted at 2026-09-25 12:17:59 on dist-test-slave-clmx"
server_key: "6bc940d93358af3686cd4fedc6a8ae1e"
server_key_iv: "bedd639b95322d5a3cb62055e401bf9a"
server_key_version: "encryptionkey@0"
I20260925 12:17:59.377801 10438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:17:59.380862 10438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:17:59.381073 10438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:17:59.381192 10438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:17:59.381379 10438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:17:59.381629 10438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:17:59.381676 10438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:59.381708 10438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:17:59.381728 10438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:17:59.386842 10438 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37815
I20260925 12:17:59.386888 10565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37815 every 8 connection(s)
I20260925 12:17:59.387125 10438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:17:59.389449 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:59.387689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60561 (local address 127.0.98.62:41951)
0925 12:17:59.387748 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:59.387750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:59.387946 (+ 196us) server_negotiation.cc:407] Connection header received
0925 12:17:59.388272 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:59.388273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:59.388277 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:59.388323 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:59.389266 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:17:59.389267 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:17:59.389267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:17:59.389271 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:17:59.389283 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:17:59.389288 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:17:59.389341 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 12:17:59.389387 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 12:17:59.389400 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 12:17:59.389616 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:17:59.387690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41951 (local address 127.0.98.3:60561)
0925 12:17:59.387848 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 12:17:59.387876 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 12:17:59.388199 (+ 323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:17:59.388362 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:17:59.388370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:17:59.388377 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:17:59.389136 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:17:59.389136 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:17:59.389157 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:17:59.389178 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:17:59.389338 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:17:59.389340 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:17:59.389342 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:17:59.389393 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 12:17:59.389431 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":74,"threads_started":1}
I20260925 12:17:59.390050 10566 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41951
I20260925 12:17:59.390159 10566 heartbeater.cc:464] Registering TS with master...
I20260925 12:17:59.390285 10566 heartbeater.cc:511] Master 127.0.98.62:41951 requested a full tablet report, sending...
I20260925 12:17:59.390614 10116 ts_manager.cc:194] Registered new tserver with Master: 0bfcc5bb2e88449a8c8665afc312f505 (127.0.98.3:37815)
I20260925 12:17:59.391955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 10438
I20260925 12:17:59.392053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:17:59.392213 392 external_mini_cluster.cc:1428] Setting key 41e36af31972851cace765c7ec828434
I20260925 12:17:59.392961 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 12:17:59.394506 10164 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:17:59.394106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.0.98.62:41951)
0925 12:17:59.394156 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 12:17:59.394157 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:17:59.394222 (+ 65us) server_negotiation.cc:407] Connection header received
0925 12:17:59.394276 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:17:59.394277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:17:59.394280 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:17:59.394317 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:17:59.394450 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44534: BlockingRecv error: recv got EOF from 127.0.0.1:44534 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:17:59.394594 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 10176
I20260925 12:17:59.398933 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 10307
I20260925 12:17:59.402976 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 10438
I20260925 12:17:59.407258 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 10085
2026-09-25T12:17:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (519 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:17:59 dist-test-slave-clmx krb5kdc[10576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:17:59 dist-test-slave-clmx krb5kdc[10576](info): set up 2 sockets
Sep 25 12:17:59 dist-test-slave-clmx krb5kdc[10576](info): commencing operation
krb5kdc: starting...
W20260925 12:18:00.416541 392 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:18:00 dist-test-slave-clmx krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:18:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:18:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:18:00.521713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44399
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:18:00.595222 10589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:18:00.595400 10589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:18:00.595422 10589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:18:00.596571 10589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:18:00.596633 10589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:18:00.596648 10589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:18:00.596660 10589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:18:00.596673 10589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:18:00.597955 10589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:18:00.598116 10589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:18:00.598256 10589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:18:00.602722 10594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:18:00.602727 10595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:18:00.602735 10597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:18:00.602907 10589 server_base.cc:1034] running on GCE node
I20260925 12:18:00.603157 10589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:18:00.603430 10589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:18:00.604568 10589 hybrid_clock.cc:648] HybridClock initialized: now 1790338680604539 us; error 49 us; skew 500 ppm
Sep 25 12:18:00 dist-test-slave-clmx krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:18:00.606769 10589 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:18:00.607146 10589 webserver.cc:460] Webserver started at http://127.0.98.62:40485/ using document root <none> and password file <none>
I20260925 12:18:00.607357 10589 fs_manager.cc:359] Metadata directory not provided
I20260925 12:18:00.607398 10589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:18:00.607525 10589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:18:00.608461 10589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "92150e20f64c4bcea71b8e5d3f5cb406"
format_stamp: "Formatted at 2026-09-25 12:18:00 on dist-test-slave-clmx"
server_key: "3f81a2ad0b7be433617605644af9f545"
server_key_iv: "82fd7379b3a3a16631f699d32fb909d0"
server_key_version: "encryptionkey@0"
I20260925 12:18:00.608765 10589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "92150e20f64c4bcea71b8e5d3f5cb406"
format_stamp: "Formatted at 2026-09-25 12:18:00 on dist-test-slave-clmx"
server_key: "3f81a2ad0b7be433617605644af9f545"
server_key_iv: "82fd7379b3a3a16631f699d32fb909d0"
server_key_version: "encryptionkey@0"
I20260925 12:18:00.609869 10589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:18:00.610496 10604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:00.610644 10589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:18:00.610726 10589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "92150e20f64c4bcea71b8e5d3f5cb406"
format_stamp: "Formatted at 2026-09-25 12:18:00 on dist-test-slave-clmx"
server_key: "3f81a2ad0b7be433617605644af9f545"
server_key_iv: "82fd7379b3a3a16631f699d32fb909d0"
server_key_version: "encryptionkey@0"
I20260925 12:18:00.610805 10589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:18:00.654733 10589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:18:00.656584 10589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:18:00.656720 10589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:18:00.660344 10589 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44399
I20260925 12:18:00.660401 10656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44399 every 8 connection(s)
I20260925 12:18:00.660707 10589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:18:00.661649 10657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:18:00.663745 10657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406: Bootstrap starting.
I20260925 12:18:00.664304 10657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406: Neither blocks nor log segments found. Creating new log.
I20260925 12:18:00.664587 10657 log.cc:824] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406: Log is configured to *not* fsync() on all Append() calls
I20260925 12:18:00.665217 10657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406: No bootstrap required, opened a new log
I20260925 12:18:00.665952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 10589
I20260925 12:18:00.666043 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:18:00.666225 392 external_mini_cluster.cc:1428] Setting key 15ab88872151ce194b5c2f4e60d3df6f
I20260925 12:18:00.667300 10657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92150e20f64c4bcea71b8e5d3f5cb406" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44399 } }
I20260925 12:18:00.667460 10657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:18:00.667495 10657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92150e20f64c4bcea71b8e5d3f5cb406, State: Initialized, Role: FOLLOWER
I20260925 12:18:00.667675 10657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92150e20f64c4bcea71b8e5d3f5cb406" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44399 } }
I20260925 12:18:00.667753 10657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:18:00.667774 10657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:18:00.667833 10657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:18:00.668802 10657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92150e20f64c4bcea71b8e5d3f5cb406" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44399 } }
I20260925 12:18:00.668905 10657 leader_election.cc:302] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92150e20f64c4bcea71b8e5d3f5cb406; no voters:
I20260925 12:18:00.669032 10657 leader_election.cc:288] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:18:00.669056 10662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:18:00.669224 10657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:18:00.669246 10662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406 [term 1 LEADER]: Becoming Leader. State: Replica: 92150e20f64c4bcea71b8e5d3f5cb406, State: Running, Role: LEADER
I20260925 12:18:00.669454 10662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92150e20f64c4bcea71b8e5d3f5cb406" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44399 } }
I20260925 12:18:00.669843 10663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92150e20f64c4bcea71b8e5d3f5cb406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92150e20f64c4bcea71b8e5d3f5cb406" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44399 } } }
I20260925 12:18:00.669924 10663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406 [sys.catalog]: This master's current role is: LEADER
I20260925 12:18:00.670372 10671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:18:00.670799 10664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92150e20f64c4bcea71b8e5d3f5cb406. Latest consensus state: current_term: 1 leader_uuid: "92150e20f64c4bcea71b8e5d3f5cb406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92150e20f64c4bcea71b8e5d3f5cb406" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44399 } } }
I20260925 12:18:00.670881 10664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406 [sys.catalog]: This master's current role is: LEADER
I20260925 12:18:00.670975 10671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:18:00.672624 10671 catalog_manager.cc:1347] Generated new cluster ID: 35b68a56f94143b0baeb82d9929fdec1
I20260925 12:18:00.672680 10671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:18:00 dist-test-slave-clmx krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:18:00.677285 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:00.666699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.0.98.62:44399)
0925 12:18:00.666915 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:00.666923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:00.666954 (+ 31us) server_negotiation.cc:407] Connection header received
0925 12:18:00.667267 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:00.667275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:00.667335 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:00.667477 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:18:00.668244 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:00.669538 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:00.670540 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:00.671081 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:00.671172 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:00.673633 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:18:00.673637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:18:00.673637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:18:00.673646 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:18:00.675416 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:18:00.675824 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:18:00.675844 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:18:00.675844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:18:00.675970 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:18:00.676187 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:18:00.676188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:18:00.676188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:18:00.676468 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:18:00.676944 (+ 476us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:00.677002 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 12:18:00.677116 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":119,"threads_started":1}
I20260925 12:18:00.689513 10671 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:18:00.689982 10671 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:18:00.694432 10671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406: Generated new TSK 0
I20260925 12:18:00.694610 10671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:18:00.723564 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44399
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:18:00.813900 10683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:18:00.814096 10683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:18:00.814128 10683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:18:00.815640 10683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:18:00.815688 10683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:18:00.815726 10683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:18:00.817595 10683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:18:00.817814 10683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:18:00.818013 10683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:18:00.822592 10689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:18:00.822613 10691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:18:00.822620 10683 server_base.cc:1034] running on GCE node
W20260925 12:18:00.822644 10688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:18:00.822933 10683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:18:00.823169 10683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:18:00.824311 10683 hybrid_clock.cc:648] HybridClock initialized: now 1790338680824300 us; error 26 us; skew 500 ppm
Sep 25 12:18:00 dist-test-slave-clmx krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:18:00.826613 10683 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:18:00.826978 10683 webserver.cc:460] Webserver started at http://127.0.98.1:46181/ using document root <none> and password file <none>
I20260925 12:18:00.827169 10683 fs_manager.cc:359] Metadata directory not provided
I20260925 12:18:00.827209 10683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:18:00.827353 10683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:18:00.828357 10683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "a7fc3e53d72845d796ec04b98bbbeee1"
format_stamp: "Formatted at 2026-09-25 12:18:00 on dist-test-slave-clmx"
server_key: "6c337dadaa9ecd289bb5a39ff3fbfca1"
server_key_iv: "dab9e76170ad582aff2394ac42ecb200"
server_key_version: "encryptionkey@0"
I20260925 12:18:00.828688 10683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a7fc3e53d72845d796ec04b98bbbeee1"
format_stamp: "Formatted at 2026-09-25 12:18:00 on dist-test-slave-clmx"
server_key: "6c337dadaa9ecd289bb5a39ff3fbfca1"
server_key_iv: "dab9e76170ad582aff2394ac42ecb200"
server_key_version: "encryptionkey@0"
I20260925 12:18:00.829895 10683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:18:00.830546 10698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:00.830708 10683 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:18:00.830793 10683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "a7fc3e53d72845d796ec04b98bbbeee1"
format_stamp: "Formatted at 2026-09-25 12:18:00 on dist-test-slave-clmx"
server_key: "6c337dadaa9ecd289bb5a39ff3fbfca1"
server_key_iv: "dab9e76170ad582aff2394ac42ecb200"
server_key_version: "encryptionkey@0"
I20260925 12:18:00.830878 10683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:18:00.855602 10683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:18:00.857174 10683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:18:00.857317 10683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:18:00.857501 10683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:18:00.857766 10683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:18:00.857815 10683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:00.857861 10683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:18:00.857889 10683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:00.863189 10683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37677
I20260925 12:18:00.863235 10811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37677 every 8 connection(s)
I20260925 12:18:00.863513 10683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:18:00.868173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 10683
I20260925 12:18:00.868245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:18:00.868434 392 external_mini_cluster.cc:1428] Setting key 4619578780b4e702b19f89b5d9d1d68b
Sep 25 12:18:00 dist-test-slave-clmx krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:18:00.872375 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:00.864131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44399 (local address 127.0.98.1:48089)
0925 12:18:00.864286 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 12:18:00.864322 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:18:00.864770 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:18:00.864964 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:18:00.864967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:18:00.865485 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:18:00.865648 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:00.865652 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:00.866255 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:00.866256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:00.866958 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:00.866960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:00.867289 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:00.867290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:00.867378 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:00.867811 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:18:00.867818 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:18:00.869249 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:18:00.870978 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:18:00.870983 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:18:00.870985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:18:00.871303 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:18:00.871528 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:18:00.871529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:18:00.871529 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:18:00.871675 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:18:00.871951 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:18:00.871954 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:18:00.872051 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 12:18:00.872126 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 12:18:00.872229 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260925 12:18:00.872385 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:00.864177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48089 (local address 127.0.98.62:44399)
0925 12:18:00.864220 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:00.864222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:00.864411 (+ 189us) server_negotiation.cc:407] Connection header received
0925 12:18:00.864849 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:00.864850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:00.864859 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:00.864919 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:18:00.865722 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:00.866186 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:00.867028 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:00.867212 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:00.867259 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:00.869334 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:18:00.869336 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:18:00.869336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:18:00.869342 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:18:00.870911 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:18:00.871384 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:18:00.871385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:18:00.871386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:18:00.871459 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:18:00.871743 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:18:00.871744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:18:00.871744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:18:00.871887 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:18:00.871954 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:00.872169 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 12:18:00.872255 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:18:00.872967 10812 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44399
I20260925 12:18:00.873064 10812 heartbeater.cc:464] Registering TS with master...
I20260925 12:18:00.873224 10812 heartbeater.cc:511] Master 127.0.98.62:44399 requested a full tablet report, sending...
I20260925 12:18:00.873663 10621 ts_manager.cc:194] Registered new tserver with Master: a7fc3e53d72845d796ec04b98bbbeee1 (127.0.98.1:37677)
I20260925 12:18:00.874189 10621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48089
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:18:00.890091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44399
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:18:00.964349 10817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:18:00.964521 10817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:18:00.964543 10817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:18:00.965571 10817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:18:00.965603 10817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:18:00.965628 10817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:18:00.966907 10817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:18:00.967060 10817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:18:00.967208 10817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:18:00.971860 10823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:18:00.971860 10825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:18:00.971881 10822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:18:00.971935 10817 server_base.cc:1034] running on GCE node
I20260925 12:18:00.972242 10817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:18:00.972474 10817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:18:00.973608 10817 hybrid_clock.cc:648] HybridClock initialized: now 1790338680973594 us; error 34 us; skew 500 ppm
Sep 25 12:18:00 dist-test-slave-clmx krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:18:00.975989 10817 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:18:00.976369 10817 webserver.cc:460] Webserver started at http://127.0.98.2:38507/ using document root <none> and password file <none>
I20260925 12:18:00.976583 10817 fs_manager.cc:359] Metadata directory not provided
I20260925 12:18:00.976646 10817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:18:00.976785 10817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:18:00.977737 10817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "d2ca84ad822b4def92d1597ed922b5b2"
format_stamp: "Formatted at 2026-09-25 12:18:00 on dist-test-slave-clmx"
server_key: "c06c4ecf3c04924689903a8931c3e51a"
server_key_iv: "b5d2180248d3061f0fb8603fb14c54e4"
server_key_version: "encryptionkey@0"
I20260925 12:18:00.978072 10817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d2ca84ad822b4def92d1597ed922b5b2"
format_stamp: "Formatted at 2026-09-25 12:18:00 on dist-test-slave-clmx"
server_key: "c06c4ecf3c04924689903a8931c3e51a"
server_key_iv: "b5d2180248d3061f0fb8603fb14c54e4"
server_key_version: "encryptionkey@0"
I20260925 12:18:00.979244 10817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:18:00.979954 10832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:00.980093 10817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:00.980185 10817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "d2ca84ad822b4def92d1597ed922b5b2"
format_stamp: "Formatted at 2026-09-25 12:18:00 on dist-test-slave-clmx"
server_key: "c06c4ecf3c04924689903a8931c3e51a"
server_key_iv: "b5d2180248d3061f0fb8603fb14c54e4"
server_key_version: "encryptionkey@0"
I20260925 12:18:00.980260 10817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:18:00.987385 10817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:18:00.989425 10817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:18:00.989567 10817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:18:00.989756 10817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:18:00.990010 10817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:18:00.990060 10817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:00.990096 10817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:18:00.990123 10817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:00.995453 10817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41487
I20260925 12:18:00.995522 10945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41487 every 8 connection(s)
I20260925 12:18:00.995833 10817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:18:00 dist-test-slave-clmx krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:18:01.003085 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:00.996461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33425 (local address 127.0.98.62:44399)
0925 12:18:00.996499 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:00.996501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:00.996650 (+ 149us) server_negotiation.cc:407] Connection header received
0925 12:18:00.997068 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:00.997068 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:00.997076 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:00.997126 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:18:00.997895 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:00.998323 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:00.999124 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:00.999285 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:00.999331 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:01.001015 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:18:01.001017 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:18:01.001017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:18:01.001023 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:18:01.002071 (+ 1048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:18:01.002397 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:18:01.002397 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:18:01.002398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:18:01.002443 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:18:01.002631 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:18:01.002632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:18:01.002632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:18:01.002721 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:18:01.002767 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:01.002970 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 12:18:01.003028 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 12:18:01.003206 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:00.996413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44399 (local address 127.0.98.2:33425)
0925 12:18:00.996551 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 12:18:00.996586 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:18:00.996998 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:18:00.997175 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:18:00.997178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:18:00.997666 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:18:00.997816 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:00.997819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:00.998389 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:00.998389 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:00.999062 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:00.999063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:00.999348 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:00.999349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:00.999431 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:00.999798 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:18:00.999804 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:18:01.000933 (+ 1129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:18:01.002138 (+ 1205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:18:01.002143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:18:01.002145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:18:01.002335 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:18:01.002499 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:18:01.002500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:18:01.002500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:18:01.002581 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:18:01.002771 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:18:01.002775 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:18:01.002866 (+ 91us) client_negotiation.cc:769] Sending connection context
0925 12:18:01.002946 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 12:18:01.003043 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":81,"threads_started":1}
I20260925 12:18:01.003681 10946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44399
I20260925 12:18:01.003782 10946 heartbeater.cc:464] Registering TS with master...
I20260925 12:18:01.003938 10946 heartbeater.cc:511] Master 127.0.98.62:44399 requested a full tablet report, sending...
I20260925 12:18:01.004205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 10817
I20260925 12:18:01.004277 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:18:01.004308 10621 ts_manager.cc:194] Registered new tserver with Master: d2ca84ad822b4def92d1597ed922b5b2 (127.0.98.2:41487)
I20260925 12:18:01.004422 392 external_mini_cluster.cc:1428] Setting key ea4664e5162eb86ca3ba10a31be9cf30
I20260925 12:18:01.004703 10621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33425
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:18:01.026314 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44399
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:18:01.102082 10951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:18:01.102295 10951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:18:01.102330 10951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:18:01.103415 10951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:18:01.103477 10951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:18:01.103519 10951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:18:01.104954 10951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:18:01.105201 10951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:18:01.105423 10951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:18:01.109959 10957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:18:01.110023 10951 server_base.cc:1034] running on GCE node
W20260925 12:18:01.109965 10956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:18:01.109965 10959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:18:01.110348 10951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:18:01.110590 10951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:18:01.111744 10951 hybrid_clock.cc:648] HybridClock initialized: now 1790338681111725 us; error 30 us; skew 500 ppm
Sep 25 12:18:01 dist-test-slave-clmx krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:18:01.114120 10951 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:18:01.114514 10951 webserver.cc:460] Webserver started at http://127.0.98.3:42923/ using document root <none> and password file <none>
I20260925 12:18:01.114673 10951 fs_manager.cc:359] Metadata directory not provided
I20260925 12:18:01.114712 10951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:18:01.114812 10951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:18:01.115793 10951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "3d1b4d04fa1a4ff797cfcc36aa1317ad"
format_stamp: "Formatted at 2026-09-25 12:18:01 on dist-test-slave-clmx"
server_key: "5a436a116e3479b63614ebf0df72ca96"
server_key_iv: "4efe919152bb0227b47a1cedb3f1c8fa"
server_key_version: "encryptionkey@0"
I20260925 12:18:01.116091 10951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3d1b4d04fa1a4ff797cfcc36aa1317ad"
format_stamp: "Formatted at 2026-09-25 12:18:01 on dist-test-slave-clmx"
server_key: "5a436a116e3479b63614ebf0df72ca96"
server_key_iv: "4efe919152bb0227b47a1cedb3f1c8fa"
server_key_version: "encryptionkey@0"
I20260925 12:18:01.117251 10951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:18:01.117868 10966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:01.118007 10951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:18:01.118083 10951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "3d1b4d04fa1a4ff797cfcc36aa1317ad"
format_stamp: "Formatted at 2026-09-25 12:18:01 on dist-test-slave-clmx"
server_key: "5a436a116e3479b63614ebf0df72ca96"
server_key_iv: "4efe919152bb0227b47a1cedb3f1c8fa"
server_key_version: "encryptionkey@0"
I20260925 12:18:01.118131 10951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:18:01.135300 10951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:18:01.136910 10951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:18:01.137010 10951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:18:01.137182 10951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:18:01.137435 10951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:18:01.137476 10951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:01.137494 10951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:18:01.137506 10951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:01.142621 10951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41489
I20260925 12:18:01.142680 11079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41489 every 8 connection(s)
I20260925 12:18:01.142949 10951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 12:18:01 dist-test-slave-clmx krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:18:01.150723 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:01.143551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48859 (local address 127.0.98.62:44399)
0925 12:18:01.143590 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:01.143592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:01.143916 (+ 324us) server_negotiation.cc:407] Connection header received
0925 12:18:01.144361 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:01.144362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:01.144377 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:01.144427 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:18:01.145254 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.145730 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.146642 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.146806 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.146850 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:01.148657 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:18:01.148658 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:18:01.148659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:18:01.148665 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:18:01.149763 (+ 1098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:18:01.150073 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:18:01.150074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:18:01.150074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:18:01.150114 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:18:01.150293 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:18:01.150294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:18:01.150294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:18:01.150381 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:18:01.150424 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:01.150615 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 12:18:01.150668 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:18:01.150851 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:01.143540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44399 (local address 127.0.98.3:48859)
0925 12:18:01.143759 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 12:18:01.143810 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 12:18:01.144282 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:18:01.144496 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:18:01.144499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:18:01.145020 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:18:01.145184 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:01.145188 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.145799 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.145799 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:01.146564 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:01.146566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.146874 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.146875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:01.146962 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:01.147317 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:18:01.147324 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:18:01.148559 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:18:01.149825 (+ 1266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:18:01.149830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:18:01.149832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:18:01.150016 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:18:01.150166 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:18:01.150166 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:18:01.150166 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:18:01.150246 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:18:01.150429 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:18:01.150432 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:18:01.150520 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 12:18:01.150590 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 12:18:01.150685 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":124,"threads_started":1}
I20260925 12:18:01.150972 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 10951
I20260925 12:18:01.151099 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:18:01.151288 392 external_mini_cluster.cc:1428] Setting key 7069403b441e539c1c3ec1daf558e0bc
I20260925 12:18:01.151296 11080 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44399
I20260925 12:18:01.151392 11080 heartbeater.cc:464] Registering TS with master...
I20260925 12:18:01.151589 11080 heartbeater.cc:511] Master 127.0.98.62:44399 requested a full tablet report, sending...
I20260925 12:18:01.151957 10621 ts_manager.cc:194] Registered new tserver with Master: 3d1b4d04fa1a4ff797cfcc36aa1317ad (127.0.98.3:41489)
I20260925 12:18:01.152345 10621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48859
I20260925 12:18:01.163105 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:18:01.169584 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:01.164360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49756 (local address 127.0.98.62:44399)
0925 12:18:01.164409 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:01.164411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:01.164497 (+ 86us) server_negotiation.cc:407] Connection header received
0925 12:18:01.164539 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:01.164540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:01.164547 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:01.164592 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:18:01.165128 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.165570 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.166264 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.166428 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.166492 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:01.167427 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:18:01.167428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:18:01.167429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:18:01.167434 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:18:01.168462 (+ 1028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:18:01.168782 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:18:01.168783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:18:01.168784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:18:01.168831 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:18:01.169035 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:18:01.169036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:18:01.169036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:18:01.169214 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:18:01.169306 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:01.169435 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 12:18:01.169499 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 12:18:01.170900 10621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:18:01.172436 10621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:18:01.178249 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:01.174266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41487 (local address 127.0.0.1:52074)
0925 12:18:01.174362 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 12:18:01.174395 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 12:18:01.174449 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:18:01.174760 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:18:01.174764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:18:01.174766 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:18:01.174822 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:01.174823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.175926 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.175927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:01.177260 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:01.177261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.177988 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.177989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:01.178062 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:01.178073 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 12:18:01.178141 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 12:18:01.178186 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":56,"threads_started":1}
I20260925 12:18:01.178269 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:01.174350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52074 (local address 127.0.98.2:41487)
0925 12:18:01.174514 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:01.174519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:01.174530 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:18:01.174569 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:01.174571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:01.174586 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:01.174658 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:18:01.174927 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.175813 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.177353 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.177907 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.177990 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:01.178043 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:01.178154 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 12:18:01.178194 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260925 12:18:01.178409 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:01.174457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41489 (local address 127.0.0.1:34420)
0925 12:18:01.174530 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 12:18:01.174540 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:18:01.174590 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:18:01.175022 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:18:01.175023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:18:01.175025 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:18:01.175072 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:01.175074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.176329 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.176330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:01.177350 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:01.177352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.178194 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.178195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:01.178256 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:01.178260 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:18:01.178289 (+ 29us) client_negotiation.cc:240] Negotiation successful
0925 12:18:01.178329 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"thread_start_us":37,"threads_started":1}
I20260925 12:18:01.178584 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:01.174567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37677 (local address 127.0.0.1:59912)
0925 12:18:01.174852 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 12:18:01.174863 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:18:01.174918 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:18:01.175235 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:18:01.175236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:18:01.175238 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:18:01.175267 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:01.175268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.176511 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.176512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:01.177499 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:01.177502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.178393 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.178394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:01.178450 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:01.178453 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:18:01.178499 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 12:18:01.178532 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":46,"threads_started":1}
I20260925 12:18:01.178804 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:01.174673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59912 (local address 127.0.98.1:37677)
0925 12:18:01.174852 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:01.174857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:01.174907 (+ 50us) server_negotiation.cc:407] Connection header received
0925 12:18:01.174982 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:01.174984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:01.174999 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:01.175161 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:18:01.175357 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.176420 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.177578 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.178257 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.178315 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:01.178363 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:01.178681 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 12:18:01.178731 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
I20260925 12:18:01.179077 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:01.174711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34420 (local address 127.0.98.3:41489)
0925 12:18:01.174883 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:01.174887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:01.174897 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:18:01.174928 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:01.174930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:01.174943 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:01.175014 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:18:01.175150 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.176224 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.177434 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.178095 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.178859 (+ 764us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:01.178923 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:01.178966 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 12:18:01.179002 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
I20260925 12:18:01.179270 10880 tablet_service.cc:1467] Processing CreateTablet for tablet 0235b7a6885f4489a25d1efae22b2b96 (DEFAULT_TABLE table=test-table [id=c1813da877124fdebc8cc8d990dea151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:18:01.179381 10746 tablet_service.cc:1467] Processing CreateTablet for tablet 0235b7a6885f4489a25d1efae22b2b96 (DEFAULT_TABLE table=test-table [id=c1813da877124fdebc8cc8d990dea151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:18:01.179639 10880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0235b7a6885f4489a25d1efae22b2b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:18:01.179675 10746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0235b7a6885f4489a25d1efae22b2b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:18:01.179612 11014 tablet_service.cc:1467] Processing CreateTablet for tablet 0235b7a6885f4489a25d1efae22b2b96 (DEFAULT_TABLE table=test-table [id=c1813da877124fdebc8cc8d990dea151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:18:01.179949 11014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0235b7a6885f4489a25d1efae22b2b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:18:01.182001 11094 tablet_bootstrap.cc:492] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad: Bootstrap starting.
I20260925 12:18:01.182756 11094 tablet_bootstrap.cc:654] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad: Neither blocks nor log segments found. Creating new log.
I20260925 12:18:01.182747 11096 tablet_bootstrap.cc:492] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2: Bootstrap starting.
I20260925 12:18:01.183297 11094 log.cc:824] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad: Log is configured to *not* fsync() on all Append() calls
I20260925 12:18:01.183362 11096 tablet_bootstrap.cc:654] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2: Neither blocks nor log segments found. Creating new log.
I20260925 12:18:01.183934 11096 log.cc:824] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2: Log is configured to *not* fsync() on all Append() calls
I20260925 12:18:01.184114 11094 tablet_bootstrap.cc:492] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad: No bootstrap required, opened a new log
I20260925 12:18:01.184186 11094 ts_tablet_manager.cc:1397] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:18:01.184245 11095 tablet_bootstrap.cc:492] T 0235b7a6885f4489a25d1efae22b2b96 P a7fc3e53d72845d796ec04b98bbbeee1: Bootstrap starting.
I20260925 12:18:01.184834 11095 tablet_bootstrap.cc:654] T 0235b7a6885f4489a25d1efae22b2b96 P a7fc3e53d72845d796ec04b98bbbeee1: Neither blocks nor log segments found. Creating new log.
I20260925 12:18:01.185345 11095 log.cc:824] T 0235b7a6885f4489a25d1efae22b2b96 P a7fc3e53d72845d796ec04b98bbbeee1: Log is configured to *not* fsync() on all Append() calls
I20260925 12:18:01.185863 11096 tablet_bootstrap.cc:492] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2: No bootstrap required, opened a new log
I20260925 12:18:01.185933 11096 ts_tablet_manager.cc:1397] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 12:18:01.186139 11095 tablet_bootstrap.cc:492] T 0235b7a6885f4489a25d1efae22b2b96 P a7fc3e53d72845d796ec04b98bbbeee1: No bootstrap required, opened a new log
I20260925 12:18:01.186206 11095 ts_tablet_manager.cc:1397] T 0235b7a6885f4489a25d1efae22b2b96 P a7fc3e53d72845d796ec04b98bbbeee1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:18:01.186307 11094 raft_consensus.cc:348] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ca84ad822b4def92d1597ed922b5b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41487 } } peers { permanent_uuid: "3d1b4d04fa1a4ff797cfcc36aa1317ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41489 } } peers { permanent_uuid: "a7fc3e53d72845d796ec04b98bbbeee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37677 } }
I20260925 12:18:01.186476 11094 raft_consensus.cc:374] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:18:01.186509 11094 raft_consensus.cc:729] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1b4d04fa1a4ff797cfcc36aa1317ad, State: Initialized, Role: FOLLOWER
I20260925 12:18:01.186652 11094 consensus_queue.cc:260] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ca84ad822b4def92d1597ed922b5b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41487 } } peers { permanent_uuid: "3d1b4d04fa1a4ff797cfcc36aa1317ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41489 } } peers { permanent_uuid: "a7fc3e53d72845d796ec04b98bbbeee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37677 } }
I20260925 12:18:01.186914 11094 ts_tablet_manager.cc:1428] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:18:01.187023 11080 heartbeater.cc:503] Master 127.0.98.62:44399 was elected leader, sending a full tablet report...
I20260925 12:18:01.187707 11096 raft_consensus.cc:348] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ca84ad822b4def92d1597ed922b5b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41487 } } peers { permanent_uuid: "3d1b4d04fa1a4ff797cfcc36aa1317ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41489 } } peers { permanent_uuid: "a7fc3e53d72845d796ec04b98bbbeee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37677 } }
I20260925 12:18:01.187855 11096 raft_consensus.cc:374] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:18:01.187889 11096 raft_consensus.cc:729] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ca84ad822b4def92d1597ed922b5b2, State: Initialized, Role: FOLLOWER
I20260925 12:18:01.188002 11096 consensus_queue.cc:260] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ca84ad822b4def92d1597ed922b5b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41487 } } peers { permanent_uuid: "3d1b4d04fa1a4ff797cfcc36aa1317ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41489 } } peers { permanent_uuid: "a7fc3e53d72845d796ec04b98bbbeee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37677 } }
I20260925 12:18:01.188097 11095 raft_consensus.cc:348] T 0235b7a6885f4489a25d1efae22b2b96 P a7fc3e53d72845d796ec04b98bbbeee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ca84ad822b4def92d1597ed922b5b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41487 } } peers { permanent_uuid: "3d1b4d04fa1a4ff797cfcc36aa1317ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41489 } } peers { permanent_uuid: "a7fc3e53d72845d796ec04b98bbbeee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37677 } }
I20260925 12:18:01.188249 11096 ts_tablet_manager.cc:1428] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260925 12:18:01.188267 11095 raft_consensus.cc:374] T 0235b7a6885f4489a25d1efae22b2b96 P a7fc3e53d72845d796ec04b98bbbeee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:18:01.188298 10946 heartbeater.cc:503] Master 127.0.98.62:44399 was elected leader, sending a full tablet report...
I20260925 12:18:01.188303 11095 raft_consensus.cc:729] T 0235b7a6885f4489a25d1efae22b2b96 P a7fc3e53d72845d796ec04b98bbbeee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7fc3e53d72845d796ec04b98bbbeee1, State: Initialized, Role: FOLLOWER
I20260925 12:18:01.188398 11095 consensus_queue.cc:260] T 0235b7a6885f4489a25d1efae22b2b96 P a7fc3e53d72845d796ec04b98bbbeee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ca84ad822b4def92d1597ed922b5b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41487 } } peers { permanent_uuid: "3d1b4d04fa1a4ff797cfcc36aa1317ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41489 } } peers { permanent_uuid: "a7fc3e53d72845d796ec04b98bbbeee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37677 } }
I20260925 12:18:01.188622 11095 ts_tablet_manager.cc:1428] T 0235b7a6885f4489a25d1efae22b2b96 P a7fc3e53d72845d796ec04b98bbbeee1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:18:01.188658 10812 heartbeater.cc:503] Master 127.0.98.62:44399 was elected leader, sending a full tablet report...
I20260925 12:18:01.225242 11101 raft_consensus.cc:482] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:18:01.225313 11101 raft_consensus.cc:504] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ca84ad822b4def92d1597ed922b5b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41487 } } peers { permanent_uuid: "3d1b4d04fa1a4ff797cfcc36aa1317ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41489 } } peers { permanent_uuid: "a7fc3e53d72845d796ec04b98bbbeee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37677 } }
I20260925 12:18:01.225536 11101 leader_election.cc:288] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1b4d04fa1a4ff797cfcc36aa1317ad (127.0.98.3:41489), a7fc3e53d72845d796ec04b98bbbeee1 (127.0.98.1:37677)
I20260925 12:18:01.228315 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:01.225694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44663 (local address 127.0.98.3:41489)
0925 12:18:01.225730 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:01.225733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:01.225778 (+ 45us) server_negotiation.cc:407] Connection header received
0925 12:18:01.225926 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:01.225927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:01.225937 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:01.226007 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:18:01.226157 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.226648 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.227529 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.227964 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.228046 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:01.228054 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:01.228228 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 12:18:01.228259 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:18:01.228320 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:01.225677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41489 (local address 127.0.98.2:44663)
0925 12:18:01.225730 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 12:18:01.225741 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:18:01.225793 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:18:01.226010 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:18:01.226011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:18:01.226012 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:18:01.226082 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:01.226084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.226727 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.226727 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:01.227436 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:01.227437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.228045 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.228046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:01.228145 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:01.228148 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:18:01.228221 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 12:18:01.228260 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 12:18:01.228389 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:01.225863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39033 (local address 127.0.98.1:37677)
0925 12:18:01.225920 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:01.225922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:01.226006 (+ 84us) server_negotiation.cc:407] Connection header received
0925 12:18:01.226084 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:01.226085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:01.226095 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:01.226149 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:18:01.226289 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.226714 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.227653 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.228023 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.228110 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:01.228131 (+ 21us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:01.228292 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 12:18:01.228322 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 12:18:01.228470 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:01.225794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37677 (local address 127.0.98.2:39033)
0925 12:18:01.225934 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 12:18:01.225943 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:18:01.226017 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:18:01.226153 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:18:01.226154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:18:01.226155 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:18:01.226218 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:01.226221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.226800 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.226800 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:01.227549 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:01.227551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.228115 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.228128 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:01.228210 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:01.228213 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:18:01.228284 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 12:18:01.228319 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260925 12:18:01.228864 10766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0235b7a6885f4489a25d1efae22b2b96" candidate_uuid: "d2ca84ad822b4def92d1597ed922b5b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7fc3e53d72845d796ec04b98bbbeee1" is_pre_election: true
I20260925 12:18:01.228864 11034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0235b7a6885f4489a25d1efae22b2b96" candidate_uuid: "d2ca84ad822b4def92d1597ed922b5b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1b4d04fa1a4ff797cfcc36aa1317ad" is_pre_election: true
I20260925 12:18:01.228994 11034 raft_consensus.cc:2454] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2ca84ad822b4def92d1597ed922b5b2 in term 0.
I20260925 12:18:01.228994 10766 raft_consensus.cc:2454] T 0235b7a6885f4489a25d1efae22b2b96 P a7fc3e53d72845d796ec04b98bbbeee1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2ca84ad822b4def92d1597ed922b5b2 in term 0.
I20260925 12:18:01.229172 10834 leader_election.cc:302] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1b4d04fa1a4ff797cfcc36aa1317ad, d2ca84ad822b4def92d1597ed922b5b2; no voters:
I20260925 12:18:01.229277 11101 raft_consensus.cc:2781] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:18:01.229327 11101 raft_consensus.cc:482] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:18:01.229359 11101 raft_consensus.cc:3037] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:18:01.229872 11101 raft_consensus.cc:504] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ca84ad822b4def92d1597ed922b5b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41487 } } peers { permanent_uuid: "3d1b4d04fa1a4ff797cfcc36aa1317ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41489 } } peers { permanent_uuid: "a7fc3e53d72845d796ec04b98bbbeee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37677 } }
I20260925 12:18:01.229983 11101 leader_election.cc:288] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1b4d04fa1a4ff797cfcc36aa1317ad (127.0.98.3:41489), a7fc3e53d72845d796ec04b98bbbeee1 (127.0.98.1:37677)
I20260925 12:18:01.230182 11034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0235b7a6885f4489a25d1efae22b2b96" candidate_uuid: "d2ca84ad822b4def92d1597ed922b5b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1b4d04fa1a4ff797cfcc36aa1317ad"
I20260925 12:18:01.230237 11034 raft_consensus.cc:3037] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:18:01.230780 11034 raft_consensus.cc:2454] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2ca84ad822b4def92d1597ed922b5b2 in term 1.
I20260925 12:18:01.230892 10766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0235b7a6885f4489a25d1efae22b2b96" candidate_uuid: "d2ca84ad822b4def92d1597ed922b5b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7fc3e53d72845d796ec04b98bbbeee1"
I20260925 12:18:01.230942 10834 leader_election.cc:302] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1b4d04fa1a4ff797cfcc36aa1317ad, d2ca84ad822b4def92d1597ed922b5b2; no voters:
I20260925 12:18:01.230973 10766 raft_consensus.cc:3037] T 0235b7a6885f4489a25d1efae22b2b96 P a7fc3e53d72845d796ec04b98bbbeee1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:18:01.231560 10766 raft_consensus.cc:2454] T 0235b7a6885f4489a25d1efae22b2b96 P a7fc3e53d72845d796ec04b98bbbeee1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2ca84ad822b4def92d1597ed922b5b2 in term 1.
I20260925 12:18:01.231701 11101 raft_consensus.cc:2781] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:18:01.231863 11101 raft_consensus.cc:686] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [term 1 LEADER]: Becoming Leader. State: Replica: d2ca84ad822b4def92d1597ed922b5b2, State: Running, Role: LEADER
I20260925 12:18:01.231943 11101 consensus_queue.cc:237] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ca84ad822b4def92d1597ed922b5b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41487 } } peers { permanent_uuid: "3d1b4d04fa1a4ff797cfcc36aa1317ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41489 } } peers { permanent_uuid: "a7fc3e53d72845d796ec04b98bbbeee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37677 } }
I20260925 12:18:01.232613 10620 catalog_manager.cc:5495] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2ca84ad822b4def92d1597ed922b5b2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d2ca84ad822b4def92d1597ed922b5b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ca84ad822b4def92d1597ed922b5b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d1b4d04fa1a4ff797cfcc36aa1317ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7fc3e53d72845d796ec04b98bbbeee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37677 } health_report { overall_health: UNKNOWN } } }
W20260925 12:18:01.246392 10947 tablet.cc:2369] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:18:01.250295 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:01.248411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52076 (local address 127.0.98.2:41487)
0925 12:18:01.248451 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:01.248452 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:01.248459 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:18:01.248501 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:01.248502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:01.248515 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:01.248564 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:18:01.248653 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.249050 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.249717 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:01.249883 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:01.249929 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:01.250051 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:18:01.250124 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:18:01.250173 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:01.250188 (+ 15us) server_negotiation.cc:299] Negotiation successful
0925 12:18:01.250202 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:18:01.251688 10766 raft_consensus.cc:1261] T 0235b7a6885f4489a25d1efae22b2b96 P a7fc3e53d72845d796ec04b98bbbeee1 [term 1 FOLLOWER]: Refusing update from remote peer d2ca84ad822b4def92d1597ed922b5b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:18:01.251709 11034 raft_consensus.cc:1261] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad [term 1 FOLLOWER]: Refusing update from remote peer d2ca84ad822b4def92d1597ed922b5b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:18:01.251960 11101 consensus_queue.cc:1035] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7fc3e53d72845d796ec04b98bbbeee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:18:01.252110 11104 consensus_queue.cc:1035] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1b4d04fa1a4ff797cfcc36aa1317ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:18:01.254068 11107 mvcc.cc:205] Tried to move back new op lower bound from 7333227238404763648 to 7333227238326546432. Current Snapshot: MvccSnapshot[applied={T|T < 7333227238404763648}]
I20260925 12:18:01.254220 11109 mvcc.cc:205] Tried to move back new op lower bound from 7333227238404763648 to 7333227238326546432. Current Snapshot: MvccSnapshot[applied={T|T < 7333227238404763648}]
I20260925 12:18:01.256153 10621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:18:01.256232 10621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:18:01.256827 10621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92150e20f64c4bcea71b8e5d3f5cb406: Sending DeleteTablet for 3 replicas of tablet 0235b7a6885f4489a25d1efae22b2b96
I20260925 12:18:01.257114 10880 tablet_service.cc:1514] Processing DeleteTablet for tablet 0235b7a6885f4489a25d1efae22b2b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:18:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52074
I20260925 12:18:01.257194 11014 tablet_service.cc:1514] Processing DeleteTablet for tablet 0235b7a6885f4489a25d1efae22b2b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:18:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34420
I20260925 12:18:01.257247 10746 tablet_service.cc:1514] Processing DeleteTablet for tablet 0235b7a6885f4489a25d1efae22b2b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:18:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59912
I20260925 12:18:01.257359 11118 tablet_replica.cc:330] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad: stopping tablet replica
I20260925 12:18:01.257452 11118 raft_consensus.cc:2229] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:18:01.257476 11119 tablet_replica.cc:330] T 0235b7a6885f4489a25d1efae22b2b96 P a7fc3e53d72845d796ec04b98bbbeee1: stopping tablet replica
I20260925 12:18:01.257535 11117 tablet_replica.cc:330] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2: stopping tablet replica
I20260925 12:18:01.257545 11118 raft_consensus.cc:2258] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:18:01.257612 11117 raft_consensus.cc:2229] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:18:01.257611 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 10683
I20260925 12:18:01.257696 11117 raft_consensus.cc:2258] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:18:01.258328 11118 ts_tablet_manager.cc:1905] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:18:01.259287 11117 ts_tablet_manager.cc:1905] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:18:01.260104 11118 ts_tablet_manager.cc:1918] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:18:01.260198 11118 log.cc:1196] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/0235b7a6885f4489a25d1efae22b2b96
I20260925 12:18:01.260435 11118 ts_tablet_manager.cc:1939] T 0235b7a6885f4489a25d1efae22b2b96 P 3d1b4d04fa1a4ff797cfcc36aa1317ad: Deleting consensus metadata
I20260925 12:18:01.260792 10606 catalog_manager.cc:4841] TS 3d1b4d04fa1a4ff797cfcc36aa1317ad (127.0.98.3:41489): tablet 0235b7a6885f4489a25d1efae22b2b96 (table test-table [id=c1813da877124fdebc8cc8d990dea151]) successfully deleted
I20260925 12:18:01.261055 11117 ts_tablet_manager.cc:1918] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:18:01.261124 11117 log.cc:1196] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/0235b7a6885f4489a25d1efae22b2b96
I20260925 12:18:01.261359 11117 ts_tablet_manager.cc:1939] T 0235b7a6885f4489a25d1efae22b2b96 P d2ca84ad822b4def92d1597ed922b5b2: Deleting consensus metadata
I20260925 12:18:01.261768 10606 catalog_manager.cc:4841] TS d2ca84ad822b4def92d1597ed922b5b2 (127.0.98.2:41487): tablet 0235b7a6885f4489a25d1efae22b2b96 (table test-table [id=c1813da877124fdebc8cc8d990dea151]) successfully deleted
W20260925 12:18:01.263166 10606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37677 (error 108)
I20260925 12:18:01.263396 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 10817
I20260925 12:18:01.263567 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:01.263450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37677 (local address 127.0.0.1:59916)
0925 12:18:01.263506 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 12:18:01.263536 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 12:18:01.264827 10606 catalog_manager.cc:4584] TS a7fc3e53d72845d796ec04b98bbbeee1 (127.0.98.1:37677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0235b7a6885f4489a25d1efae22b2b96: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37677: connect: Connection refused (error 111)
I20260925 12:18:01.268194 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 10951
I20260925 12:18:01.272343 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 10589
2026-09-25T12:18:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1866 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:18:01 dist-test-slave-clmx krb5kdc[11123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:18:01 dist-test-slave-clmx krb5kdc[11123](info): set up 2 sockets
Sep 25 12:18:01 dist-test-slave-clmx krb5kdc[11123](info): commencing operation
krb5kdc: starting...
W20260925 12:18:02.284629 392 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:18:02 dist-test-slave-clmx krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:18:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:18:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260925 12:18:02.425508 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskKTlv5c/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/postgres -l logfile start
2026-09-25 12:18:04.622 UTC [11153] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 12:18:04.622 UTC [11153] LOG: listening on IPv4 address "127.0.98.20", port 39881
2026-09-25 12:18:04.626 UTC [11153] LOG: listening on Unix socket "/tmp/.s.PGSQL.39881"
2026-09-25 12:18:04.647 UTC [11156] LOG: database system was shut down at 2026-09-25 12:18:03 UTC
2026-09-25 12:18:04.650 UTC [11153] LOG: database system is ready to accept connections
I20260925 12:18:05.576429 392 mini_postgres.cc:96] Postgres bound to 39881
2026-09-25 12:18:05.582 UTC [11164] FATAL: database "slave" does not exist
127.0.98.20:39881 - accepting connections
I20260925 12:18:05.583122 392 mini_ranger.cc:161] Starting Ranger...
I20260925 12:18:05.596988 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 12:18:06.004135 392 mini_ranger.cc:87] Created ranger Postgres database
I20260925 12:18:06.004196 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ranger-admin
2026-09-25 12:18:06,264 [I] DB FLAVOR :POSTGRES
2026-09-25 12:18:06,264 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 12:18:06,265 [I] Checking connection..
2026-09-25 12:18:06,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 12:18:06,504 [I] Checking connection passed.
2026-09-25 12:18:06,505 [I] --------- Verifying version history table ---------
2026-09-25 12:18:06,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 12:18:06,741 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 12:18:06,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 12:18:06,950 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 12:18:06,950 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 12:18:06,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 12:18:07.165 UTC [11228] WARNING: there is no transaction in progress
2026-09-25 12:18:07,171 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 12:18:07,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 12:18:07,378 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 12:18:07,378 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 12:18:07,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 12:18:07,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 12:18:07,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-clmx.c.gcp-upstream.internal','N') ;"
2026-09-25 12:18:08,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 12:18:08,296 [I] Table x_portal_user does not exist in database ranger
2026-09-25 12:18:08,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 12:18:08,535 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 12:18:08,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 12:18:08,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 12:18:08,981 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 12:18:08,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 12:18:09.833 UTC [11336] WARNING: there is no transaction in progress
2026-09-25 12:18:09.848 UTC [11336] WARNING: there is no transaction in progress
2026-09-25 12:18:09.862 UTC [11336] WARNING: there is no transaction in progress
2026-09-25 12:18:09.875 UTC [11336] WARNING: there is no transaction in progress
2026-09-25 12:18:09.892 UTC [11336] WARNING: there is no transaction in progress
2026-09-25 12:18:09.910 UTC [11336] WARNING: there is no transaction in progress
2026-09-25 12:18:09.957 UTC [11336] WARNING: there is no transaction in progress
2026-09-25 12:18:09.962 UTC [11336] WARNING: there is no transaction in progress
2026-09-25 12:18:09.974 UTC [11336] WARNING: there is no transaction in progress
2026-09-25 12:18:09.982 UTC [11336] WARNING: there is no transaction in progress
2026-09-25 12:18:09.991 UTC [11336] WARNING: there is no transaction in progress
2026-09-25 12:18:10.002 UTC [11336] WARNING: there is no transaction in progress
2026-09-25 12:18:10.011 UTC [11336] WARNING: there is no transaction in progress
2026-09-25 12:18:10.016 UTC [11336] WARNING: there is no transaction in progress
2026-09-25 12:18:10.022 UTC [11336] WARNING: there is no transaction in progress
2026-09-25 12:18:10.027 UTC [11336] WARNING: there is no transaction in progress
2026-09-25 12:18:10,570 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 12:18:10,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 12:18:10,773 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 12:18:10,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 12:18:11,013 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 12:18:11,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 12:18:11,249 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 12:18:11,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 12:18:11,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 12:18:11,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-clmx.c.gcp-upstream.internal';"
2026-09-25 12:18:11,902 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 12:18:11,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 12:18:12,132 [I] DB_PATCHES have already been applied
I20260925 12:18:12.137252 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskKTlv5c/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 12:18:12.137302 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260925 12:18:12.140082 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:54033
Sep 25, 2026 12:18:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 12:18:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 12:18:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 12:18:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 12:18:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 12:18:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 12:18:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 12:18:13.204618 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.621s user 0.000s sys 0.008s
Sep 25, 2026 12:18:32 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 12:18:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 12:18:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 12:18:32 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 12:18:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 12:18:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 12:18:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 12:18:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 12:18:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 12:18:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 12:18:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 12:18:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 12:18:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 12:18:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 12:18:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 12:18:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 12:18:32 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 12:18:33 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 12:18:33.933238 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 12:18:33 dist-test-slave-clmx krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338713, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 12:18:34.007609 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 12:18:34.016053 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 12:18:34.730373 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 12:18:34.730438 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ranger-kms
I20260925 12:18:34.730450 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:39881
2026-09-25 12:18:34,895 [I] DB FLAVOR :POSTGRES
2026-09-25 12:18:34,895 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 12:18:34,895 [I] Checking connection
2026-09-25 12:18:34,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 12:18:35,116 [I] connection success
2026-09-25 12:18:35,116 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 12:18:35,116 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 12:18:35,116 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 12:18:35,327 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 12:18:35,327 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 12:18:35,327 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 12:18:35,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39881/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 12:18:35,584 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 12:18:35.993674 392 mini_ranger_kms.cc:310] Created kms service
I20260925 12:18:36.147709 392 mini_ranger_kms.cc:326] Created kudu user
I20260925 12:18:36.186973 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 12:18:36 dist-test-slave-clmx krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338713, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:18:36.570780 392 mini_ranger_kms.cc:384] Added ranger policy
I20260925 12:18:36.570889 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 12:18:36.570919 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260925 12:18:36.573925 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11516
I20260925 12:18:36.574040 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:45513
Sep 25, 2026 12:18:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 12:18:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 12:18:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 12:18:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 12:18:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 12:18:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 12:18:38.737985 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.730s user 0.009s sys 0.139s
I20260925 12:18:38.738116 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 12:18:38.738199 392 mini_ranger_kms.cc:401] 127.0.98.20:45513/kms/v1/keys
Sep 25 12:18:38 dist-test-slave-clmx krb5kdc[11123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790338718, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 12:18:39 dist-test-slave-clmx krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338718, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 12:18:39 dist-test-slave-clmx krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338718, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 12:18:41 dist-test-slave-clmx krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:18:41.682410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41219
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41219
--ranger_config_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45513/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:18:41.763541 11567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:18:41.763798 11567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:18:41.763819 11567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:18:41.764942 11567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:18:41.764981 11567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:18:41.764995 11567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:18:41.765007 11567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:18:41.765018 11567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:18:41.766572 11567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:45513/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41219
--ranger_config_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:18:41.766796 11567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:18:41.767014 11567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:18:41.772630 11572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:18:41.772665 11573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:18:41.772773 11575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:18:41.772675 11567 server_base.cc:1034] running on GCE node
I20260925 12:18:41.773083 11567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:18:41.773386 11567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:18:41.774534 11567 hybrid_clock.cc:648] HybridClock initialized: now 1790338721774515 us; error 35 us; skew 500 ppm
Sep 25 12:18:41 dist-test-slave-clmx krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:18:41.779800 11567 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:18:41.780326 11567 webserver.cc:460] Webserver started at http://127.0.98.62:36953/ using document root <none> and password file <none>
I20260925 12:18:41.780544 11567 fs_manager.cc:359] Metadata directory not provided
I20260925 12:18:41.780582 11567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:18:41.780711 11567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 12:18:41 dist-test-slave-clmx krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:18:41.836908 11567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "f2ed2ed81df34d46b5c3cd394ac2fce9"
format_stamp: "Formatted at 2026-09-25 12:18:41 on dist-test-slave-clmx"
server_key: "e58d6c667ed8498780a0e2557d5192bd"
server_key_iv: "572fd7d7ced0244deb0e0708d1e6d0f0"
server_key_version: "kuduclusterkey@0"
I20260925 12:18:41.837357 11567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "f2ed2ed81df34d46b5c3cd394ac2fce9"
format_stamp: "Formatted at 2026-09-25 12:18:41 on dist-test-slave-clmx"
server_key: "e58d6c667ed8498780a0e2557d5192bd"
server_key_iv: "572fd7d7ced0244deb0e0708d1e6d0f0"
server_key_version: "kuduclusterkey@0"
I20260925 12:18:41.839205 11567 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 12:18:41.854573 11582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:41.854815 11567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:18:41.854925 11567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "f2ed2ed81df34d46b5c3cd394ac2fce9"
format_stamp: "Formatted at 2026-09-25 12:18:41 on dist-test-slave-clmx"
server_key: "e58d6c667ed8498780a0e2557d5192bd"
server_key_iv: "572fd7d7ced0244deb0e0708d1e6d0f0"
server_key_version: "kuduclusterkey@0"
I20260925 12:18:41.855038 11567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:18:41.869587 11567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:18:41.871778 11567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:18:41.871918 11567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:18:41.876109 11567 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41219
I20260925 12:18:41.876151 11634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41219 every 8 connection(s)
I20260925 12:18:41.876499 11567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:18:41.877146 11635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:18:41.877358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 11567
I20260925 12:18:41.877475 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:18:41.877712 392 external_mini_cluster.cc:1428] Setting key cfa7464c54f263adaa8ac87f577bb897
I20260925 12:18:41.880173 11635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9: Bootstrap starting.
I20260925 12:18:41.880847 11635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9: Neither blocks nor log segments found. Creating new log.
I20260925 12:18:41.881134 11635 log.cc:824] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9: Log is configured to *not* fsync() on all Append() calls
I20260925 12:18:41.882501 11635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9: No bootstrap required, opened a new log
I20260925 12:18:41.884738 11635 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2ed2ed81df34d46b5c3cd394ac2fce9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41219 } }
I20260925 12:18:41.884902 11635 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:18:41.884940 11635 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2ed2ed81df34d46b5c3cd394ac2fce9, State: Initialized, Role: FOLLOWER
I20260925 12:18:41.885090 11635 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2ed2ed81df34d46b5c3cd394ac2fce9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41219 } }
Sep 25 12:18:41 dist-test-slave-clmx krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:18:41.885169 11635 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:18:41.885205 11635 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:18:41.885279 11635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:18:41.885897 11635 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2ed2ed81df34d46b5c3cd394ac2fce9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41219 } }
I20260925 12:18:41.886015 11635 leader_election.cc:302] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2ed2ed81df34d46b5c3cd394ac2fce9; no voters:
I20260925 12:18:41.886186 11635 leader_election.cc:288] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:18:41.886211 11640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:18:41.886421 11640 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9 [term 1 LEADER]: Becoming Leader. State: Replica: f2ed2ed81df34d46b5c3cd394ac2fce9, State: Running, Role: LEADER
I20260925 12:18:41.886525 11640 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2ed2ed81df34d46b5c3cd394ac2fce9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41219 } }
I20260925 12:18:41.886638 11635 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:18:41.886854 11640 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2ed2ed81df34d46b5c3cd394ac2fce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2ed2ed81df34d46b5c3cd394ac2fce9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41219 } } }
I20260925 12:18:41.886945 11640 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9 [sys.catalog]: This master's current role is: LEADER
I20260925 12:18:41.887099 11640 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2ed2ed81df34d46b5c3cd394ac2fce9. Latest consensus state: current_term: 1 leader_uuid: "f2ed2ed81df34d46b5c3cd394ac2fce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2ed2ed81df34d46b5c3cd394ac2fce9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41219 } } }
I20260925 12:18:41.887195 11640 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9 [sys.catalog]: This master's current role is: LEADER
I20260925 12:18:41.889050 11635 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 12:18:41.889695 11644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:18:41.891851 11638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:41.878371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49136 (local address 127.0.98.62:41219)
0925 12:18:41.878538 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:41.878549 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:41.878579 (+ 30us) server_negotiation.cc:407] Connection header received
0925 12:18:41.879139 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:41.879146 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:41.879211 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:41.879353 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:18:41.880152 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:41.881442 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:41.882202 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:41.882765 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:41.882870 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:41.885729 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:18:41.885733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:18:41.885734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:18:41.885745 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:18:41.887753 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:18:41.888240 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:18:41.888243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:18:41.888243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:18:41.888339 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:18:41.888552 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:18:41.888553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:18:41.888554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:18:41.888848 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:18:41.889436 (+ 588us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:41.889518 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 12:18:41.889627 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260925 12:18:41.892436 11644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:18:41.894619 11644 catalog_manager.cc:1347] Generated new cluster ID: 2da5b44d34d84f589245c33f8b58c7c3
I20260925 12:18:41.894671 11644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:18:41.903530 11644 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:18:41.904114 11644 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:18:41.910168 11644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9: Generated new TSK 0
I20260925 12:18:41.910320 11644 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:18:42.564262 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41219
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:18:42.678388 11680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:18:42.678664 11680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:18:42.678695 11680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:18:42.680328 11680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:18:42.680383 11680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:18:42.680419 11680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:18:42.682351 11680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:45513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:18:42.682554 11680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:18:42.682762 11680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:18:42.688387 11686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:18:42.688441 11685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:18:42.688706 11688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:18:42.689368 11680 server_base.cc:1034] running on GCE node
I20260925 12:18:42.689577 11680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:18:42.689854 11680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:18:42.691085 11680 hybrid_clock.cc:648] HybridClock initialized: now 1790338722691065 us; error 32 us; skew 500 ppm
Sep 25 12:18:42 dist-test-slave-clmx krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:18:42.694409 11680 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:18:42.694875 11680 webserver.cc:460] Webserver started at http://127.0.98.1:41053/ using document root <none> and password file <none>
I20260925 12:18:42.695080 11680 fs_manager.cc:359] Metadata directory not provided
I20260925 12:18:42.695143 11680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:18:42.695281 11680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 12:18:42 dist-test-slave-clmx krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:18:42.708453 11680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "6f498522617d42ab83619db2642db7cf"
format_stamp: "Formatted at 2026-09-25 12:18:42 on dist-test-slave-clmx"
server_key: "a8d4949190e0c9cc389f3035f8d20444"
server_key_iv: "b32af2ab8fb3695286d61e2ee5992b80"
server_key_version: "kuduclusterkey@0"
I20260925 12:18:42.708906 11680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6f498522617d42ab83619db2642db7cf"
format_stamp: "Formatted at 2026-09-25 12:18:42 on dist-test-slave-clmx"
server_key: "a8d4949190e0c9cc389f3035f8d20444"
server_key_iv: "b32af2ab8fb3695286d61e2ee5992b80"
server_key_version: "kuduclusterkey@0"
I20260925 12:18:42.716020 11680 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.000s
I20260925 12:18:42.727022 11695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:42.727180 11680 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:18:42.727267 11680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "6f498522617d42ab83619db2642db7cf"
format_stamp: "Formatted at 2026-09-25 12:18:42 on dist-test-slave-clmx"
server_key: "a8d4949190e0c9cc389f3035f8d20444"
server_key_iv: "b32af2ab8fb3695286d61e2ee5992b80"
server_key_version: "kuduclusterkey@0"
I20260925 12:18:42.727332 11680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:18:42.742586 11680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:18:42.745003 11680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:18:42.745172 11680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:18:42.745420 11680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:18:42.745808 11680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:18:42.745894 11680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:42.745997 11680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:18:42.746068 11680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:42.753050 11680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46021
I20260925 12:18:42.753126 11810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46021 every 8 connection(s)
I20260925 12:18:42.753436 11680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:18:42.759984 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 11680
I20260925 12:18:42.760119 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
Sep 25 12:18:42 dist-test-slave-clmx krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:18:42.760301 392 external_mini_cluster.cc:1428] Setting key 82febebbbacae3e612b51a1fd2f82e6e
I20260925 12:18:42.764575 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:42.754114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41219 (local address 127.0.98.1:57769)
0925 12:18:42.754342 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 12:18:42.754370 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 12:18:42.755195 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:18:42.755599 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:18:42.755603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:18:42.755804 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:18:42.755976 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:42.755980 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:42.756714 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:42.756715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:42.757787 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:42.757790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:42.758332 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:42.758333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:42.758480 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:42.759023 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:18:42.759032 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:18:42.761133 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:18:42.762998 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:18:42.763003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:18:42.763005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:18:42.763293 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:18:42.763520 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:18:42.763521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:18:42.763521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:18:42.763666 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:18:42.763948 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:18:42.763954 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:18:42.764108 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 12:18:42.764196 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 12:18:42.764347 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":80,"threads_started":1}
I20260925 12:18:42.765216 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:42.754145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57769 (local address 127.0.98.62:41219)
0925 12:18:42.755403 (+ 1258us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:42.755405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:42.755411 (+ 6us) server_negotiation.cc:407] Connection header received
0925 12:18:42.755433 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:42.755433 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:42.755443 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:42.755527 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:18:42.756111 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:42.756626 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:42.757891 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:42.758199 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:42.758265 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:42.761226 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:18:42.761228 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:18:42.761228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:18:42.761234 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:18:42.762914 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:18:42.763386 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:18:42.763387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:18:42.763388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:18:42.763445 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:18:42.763741 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:18:42.763742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:18:42.763743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:18:42.763869 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:18:42.763925 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:42.765054 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0925 12:18:42.765128 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1232,"thread_start_us":66,"threads_started":1}
I20260925 12:18:42.765673 11811 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41219
I20260925 12:18:42.765785 11811 heartbeater.cc:464] Registering TS with master...
I20260925 12:18:42.765956 11811 heartbeater.cc:511] Master 127.0.98.62:41219 requested a full tablet report, sending...
I20260925 12:18:42.766433 11599 ts_manager.cc:194] Registered new tserver with Master: 6f498522617d42ab83619db2642db7cf (127.0.98.1:46021)
I20260925 12:18:42.767220 11599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57769
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:18:42.793759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41219
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:18:42.894284 11817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:18:42.894481 11817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:18:42.894515 11817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:18:42.895576 11817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:18:42.895646 11817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:18:42.895687 11817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:18:42.897056 11817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:45513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:18:42.897320 11817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:18:42.897542 11817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:18:42.902622 11825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:18:42.902690 11827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:18:42.902622 11824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:18:42.903402 11817 server_base.cc:1034] running on GCE node
I20260925 12:18:42.903658 11817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:18:42.903903 11817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:18:42.905076 11817 hybrid_clock.cc:648] HybridClock initialized: now 1790338722905058 us; error 35 us; skew 500 ppm
Sep 25 12:18:42 dist-test-slave-clmx krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:18:42.907433 11817 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:18:42.907912 11817 webserver.cc:460] Webserver started at http://127.0.98.2:36519/ using document root <none> and password file <none>
I20260925 12:18:42.908174 11817 fs_manager.cc:359] Metadata directory not provided
I20260925 12:18:42.908285 11817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:18:42.908473 11817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 12:18:42 dist-test-slave-clmx krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 12:18:42 dist-test-slave-clmx krb5kdc[11123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790338722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:18:42.922199 11817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "3826325428e8438aa1a0cc1cd1c3cc9f"
format_stamp: "Formatted at 2026-09-25 12:18:42 on dist-test-slave-clmx"
server_key: "167a8fe9e90d6b1b16af4b458bb71370"
server_key_iv: "8c3d6d7f6c60d0f275d85bb174b5d244"
server_key_version: "kuduclusterkey@0"
I20260925 12:18:42.922685 11817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3826325428e8438aa1a0cc1cd1c3cc9f"
format_stamp: "Formatted at 2026-09-25 12:18:42 on dist-test-slave-clmx"
server_key: "167a8fe9e90d6b1b16af4b458bb71370"
server_key_iv: "8c3d6d7f6c60d0f275d85bb174b5d244"
server_key_version: "kuduclusterkey@0"
I20260925 12:18:42.927999 11817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260925 12:18:42.950335 11834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:42.950554 11817 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.000s
I20260925 12:18:42.950635 11817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "3826325428e8438aa1a0cc1cd1c3cc9f"
format_stamp: "Formatted at 2026-09-25 12:18:42 on dist-test-slave-clmx"
server_key: "167a8fe9e90d6b1b16af4b458bb71370"
server_key_iv: "8c3d6d7f6c60d0f275d85bb174b5d244"
server_key_version: "kuduclusterkey@0"
I20260925 12:18:42.950701 11817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:18:42.962787 11817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:18:42.964875 11817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:18:42.965040 11817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:18:42.965216 11817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:18:42.965530 11817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:18:42.965574 11817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:42.965600 11817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:18:42.965667 11817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:42.972448 11817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33215
I20260925 12:18:42.972811 11817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:18:42.973245 11947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33215 every 8 connection(s)
I20260925 12:18:42.973766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 11817
I20260925 12:18:42.973860 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:18:42.974041 392 external_mini_cluster.cc:1428] Setting key 3c50a5c3c32741313c85616fa19d395a
Sep 25 12:18:42 dist-test-slave-clmx krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:18:42.983709 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:42.974228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51277 (local address 127.0.98.62:41219)
0925 12:18:42.974273 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:42.974275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:42.974284 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:18:42.975094 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:42.975095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:42.975107 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:42.975133 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:18:42.975870 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:42.976520 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:42.977861 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:42.978171 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:42.978213 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:42.980471 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:18:42.980472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:18:42.980472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:18:42.980479 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:18:42.982076 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:18:42.982607 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:18:42.982609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:18:42.982609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:18:42.982671 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:18:42.982967 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:18:42.982968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:18:42.982969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:18:42.983099 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:18:42.983181 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:42.983515 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 12:18:42.983601 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:18:42.983907 11950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:42.973390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41219 (local address 127.0.98.2:51277)
0925 12:18:42.974112 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0925 12:18:42.974152 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 12:18:42.974985 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:18:42.975290 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:18:42.975293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:18:42.975485 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:18:42.975742 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:42.975747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:42.976620 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:42.976621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:42.977763 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:42.977766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:42.978294 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:42.978295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:42.978432 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:42.978927 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:18:42.978936 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:18:42.980355 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:18:42.982172 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:18:42.982178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:18:42.982180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:18:42.982520 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:18:42.982754 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:18:42.982755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:18:42.982755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:18:42.982888 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:18:42.983180 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:18:42.983186 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:18:42.983347 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 12:18:42.983471 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 12:18:42.983609 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":82,"threads_started":1}
I20260925 12:18:42.984452 11948 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41219
I20260925 12:18:42.984565 11948 heartbeater.cc:464] Registering TS with master...
I20260925 12:18:42.984774 11948 heartbeater.cc:511] Master 127.0.98.62:41219 requested a full tablet report, sending...
I20260925 12:18:42.985131 11599 ts_manager.cc:194] Registered new tserver with Master: 3826325428e8438aa1a0cc1cd1c3cc9f (127.0.98.2:33215)
I20260925 12:18:42.985621 11599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51277
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:18:43.019752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41219
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:18:43.133080 11954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:18:43.133373 11954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:18:43.133445 11954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:18:43.135061 11954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:18:43.135113 11954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:18:43.135147 11954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:18:43.136987 11954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:45513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:18:43.137171 11954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:18:43.137363 11954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:18:43.143045 11959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:18:43.143128 11962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:18:43.143057 11960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:18:43.144079 11954 server_base.cc:1034] running on GCE node
I20260925 12:18:43.144277 11954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:18:43.144532 11954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:18:43.145713 11954 hybrid_clock.cc:648] HybridClock initialized: now 1790338723145690 us; error 37 us; skew 500 ppm
Sep 25 12:18:43 dist-test-slave-clmx krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:18:43.148537 11954 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:18:43.148965 11954 webserver.cc:460] Webserver started at http://127.0.98.3:38703/ using document root <none> and password file <none>
I20260925 12:18:43.149171 11954 fs_manager.cc:359] Metadata directory not provided
I20260925 12:18:43.149232 11954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:18:43.149355 11954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 12:18:43 dist-test-slave-clmx krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:18:43.165172 11954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "190cd4fe84704264861e87be93a39970"
format_stamp: "Formatted at 2026-09-25 12:18:43 on dist-test-slave-clmx"
server_key: "e9f85372b72de1df92dc9ceed5746f50"
server_key_iv: "21af9022eb3d8df29a9438db730d9c23"
server_key_version: "kuduclusterkey@0"
I20260925 12:18:43.166373 11954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "190cd4fe84704264861e87be93a39970"
format_stamp: "Formatted at 2026-09-25 12:18:43 on dist-test-slave-clmx"
server_key: "e9f85372b72de1df92dc9ceed5746f50"
server_key_iv: "21af9022eb3d8df29a9438db730d9c23"
server_key_version: "kuduclusterkey@0"
I20260925 12:18:43.169862 11954 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260925 12:18:43.192729 11969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:43.195809 11954 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.001s
I20260925 12:18:43.195922 11954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "190cd4fe84704264861e87be93a39970"
format_stamp: "Formatted at 2026-09-25 12:18:43 on dist-test-slave-clmx"
server_key: "e9f85372b72de1df92dc9ceed5746f50"
server_key_iv: "21af9022eb3d8df29a9438db730d9c23"
server_key_version: "kuduclusterkey@0"
I20260925 12:18:43.196033 11954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:18:43.230082 11954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:18:43.232388 11954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:18:43.232537 11954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:18:43.232801 11954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:18:43.233196 11954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:18:43.233248 11954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:43.233322 11954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:18:43.233351 11954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:18:43.243788 11954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45765
I20260925 12:18:43.243829 12082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45765 every 8 connection(s)
I20260925 12:18:43.244192 11954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:18:43.248215 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 11954
I20260925 12:18:43.248314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:18:43.248502 392 external_mini_cluster.cc:1428] Setting key c3d279589d07cbf5b8f6b6c4ff5e457a
Sep 25 12:18:43 dist-test-slave-clmx krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:18:43.255388 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:43.245093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59303 (local address 127.0.98.62:41219)
0925 12:18:43.245144 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:43.245146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:43.245360 (+ 214us) server_negotiation.cc:407] Connection header received
0925 12:18:43.245854 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:43.245855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:43.245867 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:43.245915 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:18:43.246436 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.247158 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.248665 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.248949 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.248989 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:43.251208 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:18:43.251209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:18:43.251209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:18:43.251216 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:18:43.252782 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:18:43.254385 (+ 1603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:18:43.254387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:18:43.254387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:18:43.254447 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:18:43.254700 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:18:43.254701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:18:43.254701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:18:43.254817 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:18:43.254905 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:43.255181 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 12:18:43.255292 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 12:18:43.255642 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:43.245081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41219 (local address 127.0.98.3:59303)
0925 12:18:43.245229 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 12:18:43.245267 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 12:18:43.245752 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:18:43.245984 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:18:43.245988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:18:43.246160 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:18:43.246338 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:43.246342 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.247265 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.247267 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:43.248556 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:43.248559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.249094 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.249094 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:43.249229 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:43.249757 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:18:43.249766 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:18:43.251088 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:18:43.253952 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:18:43.253958 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:18:43.253960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:18:43.254295 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:18:43.254516 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:18:43.254517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:18:43.254517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:18:43.254632 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:18:43.254883 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:18:43.254889 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:18:43.255049 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 12:18:43.255160 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 12:18:43.255358 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260925 12:18:43.256174 12083 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41219
I20260925 12:18:43.256278 12083 heartbeater.cc:464] Registering TS with master...
I20260925 12:18:43.256450 12083 heartbeater.cc:511] Master 127.0.98.62:41219 requested a full tablet report, sending...
I20260925 12:18:43.256846 11599 ts_manager.cc:194] Registered new tserver with Master: 190cd4fe84704264861e87be93a39970 (127.0.98.3:45765)
I20260925 12:18:43.257360 11599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59303
I20260925 12:18:43.260416 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:18:43.270803 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:43.263953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.0.98.62:41219)
0925 12:18:43.264009 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:43.264011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:43.264247 (+ 236us) server_negotiation.cc:407] Connection header received
0925 12:18:43.264300 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:43.264301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:43.264311 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:43.264377 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:18:43.264932 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.265469 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.266201 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.266439 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.266498 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:43.268196 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:18:43.268198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:18:43.268198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:18:43.268205 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:18:43.269763 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:18:43.270076 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:18:43.270077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:18:43.270077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:18:43.270133 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:18:43.270327 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:18:43.270328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:18:43.270328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:18:43.270435 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:18:43.270538 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:43.270634 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 12:18:43.270701 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 12:18:43.272536 11599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:18:43.274080 11599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:18:43.287071 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:43.281743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45765 (local address 127.0.0.1:46168)
0925 12:18:43.282352 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0925 12:18:43.282396 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 12:18:43.282742 (+ 346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:18:43.283032 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:18:43.283036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:18:43.283038 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:18:43.283100 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:43.283102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.284498 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.284499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:43.285878 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:43.285879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.286803 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.286804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:43.286880 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:43.286896 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 12:18:43.286950 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 12:18:43.287000 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":94,"threads_started":1}
I20260925 12:18:43.287341 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:43.281744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33215 (local address 127.0.0.1:47128)
0925 12:18:43.282850 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0925 12:18:43.282859 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:18:43.282898 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:18:43.283259 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:18:43.283260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:18:43.283261 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:18:43.283291 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:43.283292 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.284875 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.284876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:43.285877 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:43.285879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.287126 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.287127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:43.287198 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:43.287201 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:18:43.287241 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 12:18:43.287271 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1092,"mutex_wait_us":6,"thread_start_us":90,"threads_started":1}
I20260925 12:18:43.287775 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:43.282142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46168 (local address 127.0.98.3:45765)
0925 12:18:43.282304 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:43.282309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:43.282466 (+ 157us) server_negotiation.cc:407] Connection header received
0925 12:18:43.282844 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:43.282848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:43.282865 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:43.282947 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:18:43.283211 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.284421 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.285998 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.286714 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.287451 (+ 737us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:43.287528 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:43.287650 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 12:18:43.287696 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260925 12:18:43.287861 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:43.281946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47128 (local address 127.0.98.2:33215)
0925 12:18:43.283013 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:43.283018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:43.283027 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:18:43.283061 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:43.283062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:43.283075 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:43.283163 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:18:43.283496 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.284796 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.285999 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.286981 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.287031 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:43.287096 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:43.287737 (+ 641us) server_negotiation.cc:299] Negotiation successful
0925 12:18:43.287777 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1031,"thread_start_us":81,"threads_started":1}
I20260925 12:18:43.288450 11882 tablet_service.cc:1467] Processing CreateTablet for tablet e04572823b2e43769ac1d68b87a75e05 (DEFAULT_TABLE table=test-table [id=b7312560abc74b4f938d676af527381e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:18:43.288796 11882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04572823b2e43769ac1d68b87a75e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:18:43.291265 12102 tablet_bootstrap.cc:492] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f: Bootstrap starting.
I20260925 12:18:43.292025 12001 tablet_service.cc:1467] Processing CreateTablet for tablet e04572823b2e43769ac1d68b87a75e05 (DEFAULT_TABLE table=test-table [id=b7312560abc74b4f938d676af527381e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:18:43.292320 12001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04572823b2e43769ac1d68b87a75e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:18:43.292349 12102 tablet_bootstrap.cc:654] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f: Neither blocks nor log segments found. Creating new log.
I20260925 12:18:43.292691 12102 log.cc:824] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f: Log is configured to *not* fsync() on all Append() calls
I20260925 12:18:43.293649 12102 tablet_bootstrap.cc:492] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f: No bootstrap required, opened a new log
I20260925 12:18:43.293731 12102 ts_tablet_manager.cc:1397] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 12:18:43.295364 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:43.281986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46021 (local address 127.0.0.1:54858)
0925 12:18:43.283378 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0925 12:18:43.283386 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:18:43.283434 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:18:43.291915 (+ 8481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:18:43.291916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:18:43.291918 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:18:43.291950 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:43.291963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.293118 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.293119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:43.294214 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:43.294217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.295172 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.295173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:43.295230 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:43.295234 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:18:43.295263 (+ 29us) client_negotiation.cc:240] Negotiation successful
0925 12:18:43.295296 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1378,"thread_start_us":53,"threads_started":1}
I20260925 12:18:43.295732 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:43.282076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54858 (local address 127.0.98.1:46021)
0925 12:18:43.291690 (+ 9614us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:43.291695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:43.291705 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:18:43.291738 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:43.291739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:43.291754 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:43.291841 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:18:43.292035 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.293018 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.294320 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.295076 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.295520 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:43.295583 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:43.295609 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 12:18:43.295669 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9580,"thread_start_us":53,"threads_started":1}
I20260925 12:18:43.296123 12104 tablet_bootstrap.cc:492] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970: Bootstrap starting.
I20260925 12:18:43.296141 12102 raft_consensus.cc:348] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190cd4fe84704264861e87be93a39970" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45765 } } peers { permanent_uuid: "6f498522617d42ab83619db2642db7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46021 } } peers { permanent_uuid: "3826325428e8438aa1a0cc1cd1c3cc9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 } }
I20260925 12:18:43.296270 12102 raft_consensus.cc:374] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:18:43.296298 12102 raft_consensus.cc:729] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3826325428e8438aa1a0cc1cd1c3cc9f, State: Initialized, Role: FOLLOWER
I20260925 12:18:43.296267 11744 tablet_service.cc:1467] Processing CreateTablet for tablet e04572823b2e43769ac1d68b87a75e05 (DEFAULT_TABLE table=test-table [id=b7312560abc74b4f938d676af527381e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:18:43.296571 11744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04572823b2e43769ac1d68b87a75e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:18:43.297065 12104 tablet_bootstrap.cc:654] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970: Neither blocks nor log segments found. Creating new log.
I20260925 12:18:43.297176 12102 consensus_queue.cc:260] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190cd4fe84704264861e87be93a39970" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45765 } } peers { permanent_uuid: "6f498522617d42ab83619db2642db7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46021 } } peers { permanent_uuid: "3826325428e8438aa1a0cc1cd1c3cc9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 } }
I20260925 12:18:43.297343 12104 log.cc:824] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970: Log is configured to *not* fsync() on all Append() calls
I20260925 12:18:43.297434 12102 ts_tablet_manager.cc:1428] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 12:18:43.297657 11948 heartbeater.cc:503] Master 127.0.98.62:41219 was elected leader, sending a full tablet report...
I20260925 12:18:43.298020 12104 tablet_bootstrap.cc:492] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970: No bootstrap required, opened a new log
I20260925 12:18:43.298074 12104 ts_tablet_manager.cc:1397] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:18:43.299502 12107 tablet_bootstrap.cc:492] T e04572823b2e43769ac1d68b87a75e05 P 6f498522617d42ab83619db2642db7cf: Bootstrap starting.
I20260925 12:18:43.299888 12104 raft_consensus.cc:348] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190cd4fe84704264861e87be93a39970" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45765 } } peers { permanent_uuid: "6f498522617d42ab83619db2642db7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46021 } } peers { permanent_uuid: "3826325428e8438aa1a0cc1cd1c3cc9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 } }
I20260925 12:18:43.300053 12104 raft_consensus.cc:374] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:18:43.300096 12104 raft_consensus.cc:729] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190cd4fe84704264861e87be93a39970, State: Initialized, Role: FOLLOWER
I20260925 12:18:43.300197 12104 consensus_queue.cc:260] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190cd4fe84704264861e87be93a39970" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45765 } } peers { permanent_uuid: "6f498522617d42ab83619db2642db7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46021 } } peers { permanent_uuid: "3826325428e8438aa1a0cc1cd1c3cc9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 } }
I20260925 12:18:43.300408 12107 tablet_bootstrap.cc:654] T e04572823b2e43769ac1d68b87a75e05 P 6f498522617d42ab83619db2642db7cf: Neither blocks nor log segments found. Creating new log.
I20260925 12:18:43.300428 12104 ts_tablet_manager.cc:1428] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 12:18:43.300567 12083 heartbeater.cc:503] Master 127.0.98.62:41219 was elected leader, sending a full tablet report...
I20260925 12:18:43.300767 12107 log.cc:824] T e04572823b2e43769ac1d68b87a75e05 P 6f498522617d42ab83619db2642db7cf: Log is configured to *not* fsync() on all Append() calls
I20260925 12:18:43.301555 12107 tablet_bootstrap.cc:492] T e04572823b2e43769ac1d68b87a75e05 P 6f498522617d42ab83619db2642db7cf: No bootstrap required, opened a new log
I20260925 12:18:43.301635 12107 ts_tablet_manager.cc:1397] T e04572823b2e43769ac1d68b87a75e05 P 6f498522617d42ab83619db2642db7cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:18:43.304044 12107 raft_consensus.cc:348] T e04572823b2e43769ac1d68b87a75e05 P 6f498522617d42ab83619db2642db7cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190cd4fe84704264861e87be93a39970" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45765 } } peers { permanent_uuid: "6f498522617d42ab83619db2642db7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46021 } } peers { permanent_uuid: "3826325428e8438aa1a0cc1cd1c3cc9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 } }
I20260925 12:18:43.304174 12107 raft_consensus.cc:374] T e04572823b2e43769ac1d68b87a75e05 P 6f498522617d42ab83619db2642db7cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:18:43.304214 12107 raft_consensus.cc:729] T e04572823b2e43769ac1d68b87a75e05 P 6f498522617d42ab83619db2642db7cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f498522617d42ab83619db2642db7cf, State: Initialized, Role: FOLLOWER
I20260925 12:18:43.304315 12107 consensus_queue.cc:260] T e04572823b2e43769ac1d68b87a75e05 P 6f498522617d42ab83619db2642db7cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190cd4fe84704264861e87be93a39970" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45765 } } peers { permanent_uuid: "6f498522617d42ab83619db2642db7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46021 } } peers { permanent_uuid: "3826325428e8438aa1a0cc1cd1c3cc9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 } }
I20260925 12:18:43.304581 12107 ts_tablet_manager.cc:1428] T e04572823b2e43769ac1d68b87a75e05 P 6f498522617d42ab83619db2642db7cf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:18:43.304606 11811 heartbeater.cc:503] Master 127.0.98.62:41219 was elected leader, sending a full tablet report...
Sep 25 12:18:43 dist-test-slave-clmx krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:18:43.408972 12108 raft_consensus.cc:482] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:18:43.409116 12108 raft_consensus.cc:504] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190cd4fe84704264861e87be93a39970" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45765 } } peers { permanent_uuid: "6f498522617d42ab83619db2642db7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46021 } } peers { permanent_uuid: "3826325428e8438aa1a0cc1cd1c3cc9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 } }
I20260925 12:18:43.409484 12108 leader_election.cc:288] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f498522617d42ab83619db2642db7cf (127.0.98.1:46021), 3826325428e8438aa1a0cc1cd1c3cc9f (127.0.98.2:33215)
I20260925 12:18:43.413604 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:43.410328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45551 (local address 127.0.98.2:33215)
0925 12:18:43.410365 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:43.410368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:43.410486 (+ 118us) server_negotiation.cc:407] Connection header received
0925 12:18:43.410580 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:43.410581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:43.410592 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:43.410623 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:18:43.410811 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.411415 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.412748 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.413236 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.413276 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:43.413287 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:43.413494 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 12:18:43.413529 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:18:43.413604 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:43.409712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46021 (local address 127.0.98.3:43745)
0925 12:18:43.410083 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0925 12:18:43.410093 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:18:43.410136 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:18:43.410466 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:18:43.410467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:18:43.410469 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:18:43.410536 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:43.410538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.411411 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.411412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:43.412635 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:43.412639 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.413362 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.413363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:43.413439 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:43.413441 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:18:43.413483 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 12:18:43.413518 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":10}
I20260925 12:18:43.413821 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:43.410264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43745 (local address 127.0.98.1:46021)
0925 12:18:43.410310 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:43.410312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:43.410320 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:18:43.410346 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:43.410347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:43.410358 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:43.410413 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:18:43.410610 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.411331 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.412744 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.413279 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.413332 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:43.413340 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:43.413694 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 12:18:43.413745 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:18:43.414266 11765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04572823b2e43769ac1d68b87a75e05" candidate_uuid: "190cd4fe84704264861e87be93a39970" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f498522617d42ab83619db2642db7cf" is_pre_election: true
I20260925 12:18:43.414420 11765 raft_consensus.cc:2454] T e04572823b2e43769ac1d68b87a75e05 P 6f498522617d42ab83619db2642db7cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 190cd4fe84704264861e87be93a39970 in term 0.
I20260925 12:18:43.414652 11970 leader_election.cc:302] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 190cd4fe84704264861e87be93a39970, 6f498522617d42ab83619db2642db7cf; no voters:
I20260925 12:18:43.414774 12108 raft_consensus.cc:2781] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:18:43.414808 12108 raft_consensus.cc:482] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:18:43.414827 12108 raft_consensus.cc:3037] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:18:43.415788 12108 raft_consensus.cc:504] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190cd4fe84704264861e87be93a39970" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45765 } } peers { permanent_uuid: "6f498522617d42ab83619db2642db7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46021 } } peers { permanent_uuid: "3826325428e8438aa1a0cc1cd1c3cc9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 } }
I20260925 12:18:43.415900 12108 leader_election.cc:288] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [CANDIDATE]: Term 1 election: Requested vote from peers 6f498522617d42ab83619db2642db7cf (127.0.98.1:46021), 3826325428e8438aa1a0cc1cd1c3cc9f (127.0.98.2:33215)
I20260925 12:18:43.416092 11765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04572823b2e43769ac1d68b87a75e05" candidate_uuid: "190cd4fe84704264861e87be93a39970" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f498522617d42ab83619db2642db7cf"
I20260925 12:18:43.416154 11765 raft_consensus.cc:3037] T e04572823b2e43769ac1d68b87a75e05 P 6f498522617d42ab83619db2642db7cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:18:43.416860 11765 raft_consensus.cc:2454] T e04572823b2e43769ac1d68b87a75e05 P 6f498522617d42ab83619db2642db7cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 190cd4fe84704264861e87be93a39970 in term 1.
I20260925 12:18:43.417022 11970 leader_election.cc:302] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 190cd4fe84704264861e87be93a39970, 6f498522617d42ab83619db2642db7cf; no voters:
I20260925 12:18:43.417146 12108 raft_consensus.cc:2781] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:18:43.417260 12108 raft_consensus.cc:686] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [term 1 LEADER]: Becoming Leader. State: Replica: 190cd4fe84704264861e87be93a39970, State: Running, Role: LEADER
I20260925 12:18:43.417330 12108 consensus_queue.cc:237] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190cd4fe84704264861e87be93a39970" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45765 } } peers { permanent_uuid: "6f498522617d42ab83619db2642db7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46021 } } peers { permanent_uuid: "3826325428e8438aa1a0cc1cd1c3cc9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 } }
I20260925 12:18:43.418093 11599 catalog_manager.cc:5495] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 reported cstate change: term changed from 0 to 1, leader changed from <none> to 190cd4fe84704264861e87be93a39970 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "190cd4fe84704264861e87be93a39970" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190cd4fe84704264861e87be93a39970" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f498522617d42ab83619db2642db7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3826325428e8438aa1a0cc1cd1c3cc9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 } health_report { overall_health: UNKNOWN } } }
I20260925 12:18:43.418614 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:43.409814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33215 (local address 127.0.98.3:45551)
0925 12:18:43.410422 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0925 12:18:43.410432 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:18:43.410507 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:18:43.410680 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:18:43.410680 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:18:43.410681 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:18:43.410746 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:43.410747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.411517 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.411518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:43.412643 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:18:43.412644 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.413339 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.413339 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:18:43.413403 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:43.413407 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:18:43.414331 (+ 924us) client_negotiation.cc:240] Negotiation successful
0925 12:18:43.414414 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":6,"thread_start_us":73,"threads_started":1}
I20260925 12:18:43.419124 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04572823b2e43769ac1d68b87a75e05" candidate_uuid: "190cd4fe84704264861e87be93a39970" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3826325428e8438aa1a0cc1cd1c3cc9f" is_pre_election: true
I20260925 12:18:43.419265 11891 raft_consensus.cc:2454] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 190cd4fe84704264861e87be93a39970 in term 0.
I20260925 12:18:43.419364 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04572823b2e43769ac1d68b87a75e05" candidate_uuid: "190cd4fe84704264861e87be93a39970" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3826325428e8438aa1a0cc1cd1c3cc9f"
I20260925 12:18:43.419404 11892 raft_consensus.cc:3037] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:18:43.420212 11892 raft_consensus.cc:2454] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 190cd4fe84704264861e87be93a39970 in term 1.
I20260925 12:18:43.429761 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:43.426685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46172 (local address 127.0.98.3:45765)
0925 12:18:43.426727 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:18:43.426729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:18:43.426739 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:18:43.426832 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:18:43.426833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:18:43.426849 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:18:43.426920 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:18:43.427087 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.427821 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.428928 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:18:43.429195 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:18:43.429256 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:18:43.429447 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:18:43.429561 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:18:43.429637 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 12:18:43.429687 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 12:18:43.429701 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:18:43.431587 11892 raft_consensus.cc:1261] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f [term 1 FOLLOWER]: Refusing update from remote peer 190cd4fe84704264861e87be93a39970: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:18:43.431700 11765 raft_consensus.cc:1261] T e04572823b2e43769ac1d68b87a75e05 P 6f498522617d42ab83619db2642db7cf [term 1 FOLLOWER]: Refusing update from remote peer 190cd4fe84704264861e87be93a39970: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:18:43.432152 12112 consensus_queue.cc:1035] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f498522617d42ab83619db2642db7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:18:43.432358 12108 consensus_queue.cc:1035] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3826325428e8438aa1a0cc1cd1c3cc9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:18:43.434958 12115 mvcc.cc:205] Tried to move back new op lower bound from 7333227411173609472 to 7333227411117801472. Current Snapshot: MvccSnapshot[applied={T|T < 7333227411173609472}]
I20260925 12:18:43.435570 12117 mvcc.cc:205] Tried to move back new op lower bound from 7333227411173609472 to 7333227411117801472. Current Snapshot: MvccSnapshot[applied={T|T < 7333227411173609472}]
I20260925 12:18:43.437840 12116 mvcc.cc:205] Tried to move back new op lower bound from 7333227411173609472 to 7333227411117801472. Current Snapshot: MvccSnapshot[applied={T|T < 7333227411173609472}]
I20260925 12:18:43.438182 11599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:18:43.438261 11599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:18:43.439028 11599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2ed2ed81df34d46b5c3cd394ac2fce9: Sending DeleteTablet for 3 replicas of tablet e04572823b2e43769ac1d68b87a75e05
I20260925 12:18:43.439926 11744 tablet_service.cc:1514] Processing DeleteTablet for tablet e04572823b2e43769ac1d68b87a75e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:18:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54858
I20260925 12:18:43.439997 12001 tablet_service.cc:1514] Processing DeleteTablet for tablet e04572823b2e43769ac1d68b87a75e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:18:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46168
I20260925 12:18:43.440218 11882 tablet_service.cc:1514] Processing DeleteTablet for tablet e04572823b2e43769ac1d68b87a75e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:18:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47128
I20260925 12:18:43.440833 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 11680
I20260925 12:18:43.446746 12125 tablet_replica.cc:330] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970: stopping tablet replica
I20260925 12:18:43.446866 12125 raft_consensus.cc:2229] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:18:43.446955 12125 raft_consensus.cc:2258] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:18:43.447080 12126 tablet_replica.cc:330] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f: stopping tablet replica
I20260925 12:18:43.447166 12126 raft_consensus.cc:2229] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:18:43.447257 12126 raft_consensus.cc:2258] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:18:43.447685 12125 ts_tablet_manager.cc:1905] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:18:43.449573 12125 ts_tablet_manager.cc:1918] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:18:43.449645 12125 log.cc:1196] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/e04572823b2e43769ac1d68b87a75e05
I20260925 12:18:43.449895 12125 ts_tablet_manager.cc:1939] T e04572823b2e43769ac1d68b87a75e05 P 190cd4fe84704264861e87be93a39970: Deleting consensus metadata
I20260925 12:18:43.450246 11583 catalog_manager.cc:4841] TS 190cd4fe84704264861e87be93a39970 (127.0.98.3:45765): tablet e04572823b2e43769ac1d68b87a75e05 (table test-table [id=b7312560abc74b4f938d676af527381e]) successfully deleted
I20260925 12:18:43.450634 12126 ts_tablet_manager.cc:1905] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:18:43.452150 12126 ts_tablet_manager.cc:1918] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:18:43.452214 12126 log.cc:1196] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/e04572823b2e43769ac1d68b87a75e05
I20260925 12:18:43.452432 12126 ts_tablet_manager.cc:1939] T e04572823b2e43769ac1d68b87a75e05 P 3826325428e8438aa1a0cc1cd1c3cc9f: Deleting consensus metadata
I20260925 12:18:43.452800 11584 catalog_manager.cc:4841] TS 3826325428e8438aa1a0cc1cd1c3cc9f (127.0.98.2:33215): tablet e04572823b2e43769ac1d68b87a75e05 (table test-table [id=b7312560abc74b4f938d676af527381e]) successfully deleted
W20260925 12:18:43.456987 11583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46021 (error 108)
I20260925 12:18:43.457515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 11817
I20260925 12:18:43.472129 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:18:43.471977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46021 (local address 127.0.0.1:54872)
0925 12:18:43.472045 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 12:18:43.472083 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 12:18:43.472323 11583 catalog_manager.cc:4584] TS 6f498522617d42ab83619db2642db7cf (127.0.98.1:46021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e04572823b2e43769ac1d68b87a75e05: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46021: connect: Connection refused (error 111)
I20260925 12:18:43.476023 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 11954
I20260925 12:18:43.481175 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 11567
I20260925 12:18:43.486930 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 12:18:43.821382 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 12:18:43.821520 392 mini_ranger.cc:66] Stopping Ranger...
I20260925 12:18:44.166227 392 mini_ranger.cc:68] Stopped Ranger
2026-09-25 12:18:44.166 UTC [11153] LOG: received smart shutdown request
2026-09-25 12:18:44.169 UTC [11153] LOG: background worker "logical replication launcher" (PID 11162) exited with exit code 1
2026-09-25 12:18:44.169 UTC [11157] LOG: shutting down
2026-09-25 12:18:44.307 UTC [11153] LOG: database system is shut down
2026-09-25T12:18:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43142 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:18:44 dist-test-slave-clmx krb5kdc[12136](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:18:44 dist-test-slave-clmx krb5kdc[12136](info): set up 2 sockets
Sep 25 12:18:44 dist-test-slave-clmx krb5kdc[12136](info): commencing operation
krb5kdc: starting...
W20260925 12:18:45.403522 392 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:18:45 dist-test-slave-clmx krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:18:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:18:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260925 12:18:45.553951 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskKTlv5c/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/postgres -l logfile start
2026-09-25 12:18:47.766 UTC [12166] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 12:18:47.766 UTC [12166] LOG: listening on IPv4 address "127.0.98.20", port 41951
2026-09-25 12:18:47.769 UTC [12166] LOG: listening on Unix socket "/tmp/.s.PGSQL.41951"
2026-09-25 12:18:47.789 UTC [12169] LOG: database system was shut down at 2026-09-25 12:18:46 UTC
2026-09-25 12:18:47.792 UTC [12166] LOG: database system is ready to accept connections
I20260925 12:18:48.727326 392 mini_postgres.cc:96] Postgres bound to 41951
2026-09-25 12:18:48.732 UTC [12177] FATAL: database "slave" does not exist
127.0.98.20:41951 - accepting connections
I20260925 12:18:48.733448 392 mini_ranger.cc:161] Starting Ranger...
I20260925 12:18:48.746807 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 12:18:49.147066 392 mini_ranger.cc:87] Created ranger Postgres database
I20260925 12:18:49.147150 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ranger-admin
2026-09-25 12:18:49,409 [I] DB FLAVOR :POSTGRES
2026-09-25 12:18:49,409 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 12:18:49,409 [I] Checking connection..
2026-09-25 12:18:49,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 12:18:49,653 [I] Checking connection passed.
2026-09-25 12:18:49,653 [I] --------- Verifying version history table ---------
2026-09-25 12:18:49,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 12:18:49,898 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 12:18:49,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 12:18:50,143 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 12:18:50,143 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 12:18:50,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 12:18:50.364 UTC [12241] WARNING: there is no transaction in progress
2026-09-25 12:18:50,370 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 12:18:50,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 12:18:50,605 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 12:18:50,606 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 12:18:50,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 12:18:50,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 12:18:51,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-clmx.c.gcp-upstream.internal','N') ;"
2026-09-25 12:18:51,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 12:18:51,552 [I] Table x_portal_user does not exist in database ranger
2026-09-25 12:18:51,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 12:18:51,782 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 12:18:51,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 12:18:52,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 12:18:52,279 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 12:18:52,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 12:18:53.123 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 12:18:53.137 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 12:18:53.151 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 12:18:53.164 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 12:18:53.180 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 12:18:53.198 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 12:18:53.256 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 12:18:53.262 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 12:18:53.274 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 12:18:53.283 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 12:18:53.292 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 12:18:53.305 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 12:18:53.313 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 12:18:53.319 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 12:18:53.325 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 12:18:53.330 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 12:18:53,855 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 12:18:53,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 12:18:54,060 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 12:18:54,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 12:18:54,271 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 12:18:54,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 12:18:54,485 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 12:18:54,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 12:18:54,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 12:18:54,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-clmx.c.gcp-upstream.internal';"
2026-09-25 12:18:55,126 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 12:18:55,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 12:18:55,344 [I] DB_PATCHES have already been applied
I20260925 12:18:55.349742 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskKTlv5c/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 12:18:55.349797 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260925 12:18:55.352541 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:44039
Sep 25, 2026 12:18:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 12:18:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 12:18:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 12:18:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 12:18:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 12:18:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 12:18:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 12:18:56.425750 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.692s user 0.000s sys 0.008s
Sep 25, 2026 12:19:15 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 12:19:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 12:19:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 12:19:15 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 12:19:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 12:19:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 12:19:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 12:19:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 12:19:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 12:19:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 12:19:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 12:19:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 12:19:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 12:19:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 12:19:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 12:19:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 12:19:15 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 12:19:16 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 12:19:17.092787 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 12:19:17 dist-test-slave-clmx krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338757, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 12:19:17.172173 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 12:19:17.180145 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 12:19:17.963672 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 12:19:17.963753 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ranger-kms
I20260925 12:19:17.963774 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:41951
2026-09-25 12:19:18,118 [I] DB FLAVOR :POSTGRES
2026-09-25 12:19:18,118 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 12:19:18,118 [I] Checking connection
2026-09-25 12:19:18,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 12:19:18,335 [I] connection success
2026-09-25 12:19:18,335 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 12:19:18,335 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 12:19:18,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 12:19:18,586 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 12:19:18,587 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 12:19:18,587 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 12:19:18,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41951/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 12:19:18,852 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 12:19:19.270936 392 mini_ranger_kms.cc:310] Created kms service
I20260925 12:19:19.459990 392 mini_ranger_kms.cc:326] Created kudu user
I20260925 12:19:19.499157 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 12:19:19 dist-test-slave-clmx krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338757, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:19:19.892638 392 mini_ranger_kms.cc:384] Added ranger policy
I20260925 12:19:19.892756 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskKTlv5c/build/release/bin/postgresql.jar:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskKTlv5c/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 12:19:19.892776 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260925 12:19:19.895752 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12529
I20260925 12:19:19.895812 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:37217
Sep 25, 2026 12:19:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 12:19:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 12:19:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 12:19:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 12:19:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 12:19:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 12:19:21 dist-test-slave-clmx krb5kdc[12136](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790338761, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:21.995718 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.824s user 0.000s sys 0.137s
I20260925 12:19:21.995814 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 12:19:21.995857 392 mini_ranger_kms.cc:401] 127.0.98.20:37217/kms/v1/keys
Sep 25 12:19:22 dist-test-slave-clmx krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338761, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 12:19:22 dist-test-slave-clmx krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338761, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 12:19:24 dist-test-slave-clmx krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:19:24.721925 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36335
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36335
--ranger_config_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37217/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:19:24.796554 12580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:24.796718 12580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:24.796738 12580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:24.797734 12580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:19:24.797771 12580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:24.797784 12580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:24.797796 12580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:19:24.797828 12580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:19:24.799218 12580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:37217/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36335
--ranger_config_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:24.799429 12580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:24.799645 12580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:24.805281 12586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:24.805383 12588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:24.805512 12585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:24.805346 12580 server_base.cc:1034] running on GCE node
I20260925 12:19:24.805729 12580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:24.806092 12580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:24.807258 12580 hybrid_clock.cc:648] HybridClock initialized: now 1790338764807258 us; error 35 us; skew 500 ppm
Sep 25 12:19:24 dist-test-slave-clmx krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:24.809679 12580 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:19:24.810146 12580 webserver.cc:460] Webserver started at http://127.0.98.62:44181/ using document root <none> and password file <none>
I20260925 12:19:24.810343 12580 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:24.810397 12580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:24.810523 12580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 12:19:24 dist-test-slave-clmx krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:19:24.852284 12580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "912b6b185195439ab861aa3091f7ec9d"
format_stamp: "Formatted at 2026-09-25 12:19:24 on dist-test-slave-clmx"
server_key: "e1fabc7669fabc491007f5a00b792530"
server_key_iv: "385ebcb7a45fe6bee1bf8a3861581261"
server_key_version: "kuduclusterkey@0"
I20260925 12:19:24.852701 12580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "912b6b185195439ab861aa3091f7ec9d"
format_stamp: "Formatted at 2026-09-25 12:19:24 on dist-test-slave-clmx"
server_key: "e1fabc7669fabc491007f5a00b792530"
server_key_iv: "385ebcb7a45fe6bee1bf8a3861581261"
server_key_version: "kuduclusterkey@0"
I20260925 12:19:24.854031 12580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:19:24.863705 12595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:24.863884 12580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 12:19:24.863974 12580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "912b6b185195439ab861aa3091f7ec9d"
format_stamp: "Formatted at 2026-09-25 12:19:24 on dist-test-slave-clmx"
server_key: "e1fabc7669fabc491007f5a00b792530"
server_key_iv: "385ebcb7a45fe6bee1bf8a3861581261"
server_key_version: "kuduclusterkey@0"
I20260925 12:19:24.864077 12580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:24.881515 12580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:24.883437 12580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:24.883597 12580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:24.888255 12580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36335
I20260925 12:19:24.888326 12647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36335 every 8 connection(s)
I20260925 12:19:24.888646 12580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:19:24.889276 12648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:19:24.891373 12648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d: Bootstrap starting.
I20260925 12:19:24.891991 12648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d: Neither blocks nor log segments found. Creating new log.
I20260925 12:19:24.892268 12648 log.cc:824] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:24.892899 12648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d: No bootstrap required, opened a new log
I20260925 12:19:24.894857 12648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912b6b185195439ab861aa3091f7ec9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36335 } }
I20260925 12:19:24.895015 12648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:19:24.895059 12648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912b6b185195439ab861aa3091f7ec9d, State: Initialized, Role: FOLLOWER
I20260925 12:19:24.895238 12648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912b6b185195439ab861aa3091f7ec9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36335 } }
I20260925 12:19:24.895326 12648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:19:24.895367 12648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:19:24.895413 12648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:19:24.895973 12648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912b6b185195439ab861aa3091f7ec9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36335 } }
I20260925 12:19:24.896077 12648 leader_election.cc:302] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 912b6b185195439ab861aa3091f7ec9d; no voters:
I20260925 12:19:24.896258 12648 leader_election.cc:288] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:19:24.896281 12651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:19:24.896394 12651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [term 1 LEADER]: Becoming Leader. State: Replica: 912b6b185195439ab861aa3091f7ec9d, State: Running, Role: LEADER
I20260925 12:19:24.896464 12651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912b6b185195439ab861aa3091f7ec9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36335 } }
I20260925 12:19:24.896500 12648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:19:24.896500 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 12580
I20260925 12:19:24.896595 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:19:24.896754 392 external_mini_cluster.cc:1428] Setting key cbd0965c43d096633a2ddf8a21530f1a
I20260925 12:19:24.896741 12651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "912b6b185195439ab861aa3091f7ec9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912b6b185195439ab861aa3091f7ec9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36335 } } }
I20260925 12:19:24.896795 12651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [sys.catalog]: This master's current role is: LEADER
I20260925 12:19:24.896890 12651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 912b6b185195439ab861aa3091f7ec9d. Latest consensus state: current_term: 1 leader_uuid: "912b6b185195439ab861aa3091f7ec9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912b6b185195439ab861aa3091f7ec9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36335 } } }
I20260925 12:19:24.896939 12651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [sys.catalog]: This master's current role is: LEADER
I20260925 12:19:24.897068 12655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:19:24.897548 12655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:19:24.897822 12648 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 12:19:24.898939 12670 mvcc.cc:205] Tried to move back new op lower bound from 7333227581024559104 to 7333227581016215552. Current Snapshot: MvccSnapshot[applied={T|T < 7333227581024559104}]
I20260925 12:19:24.902233 12655 catalog_manager.cc:1347] Generated new cluster ID: 62bb019056ee498e8aa2aa1bf7556073
I20260925 12:19:24.902484 12655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:19:24 dist-test-slave-clmx krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:24.912441 12663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:24.897300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40500 (local address 127.0.98.62:36335)
0925 12:19:24.897805 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:24.897814 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:24.897845 (+ 31us) server_negotiation.cc:407] Connection header received
0925 12:19:24.897967 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:24.897974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:24.898027 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:24.898150 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:24.898957 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:24.902761 (+ 3804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:24.903889 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:24.904579 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:24.904685 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:24.908280 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:24.908284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:24.908284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:24.908294 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:24.910709 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:24.911047 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:24.911050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:24.911050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:24.911155 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:24.911319 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:24.911319 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:24.911320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:24.911593 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:24.911742 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:24.912116 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 12:19:24.912225 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":59,"threads_started":1}
I20260925 12:19:24.914800 12655 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:19:24.915503 12655 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:19:24.918907 12655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d: Generated new TSK 0
I20260925 12:19:24.919131 12655 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:19:25.642267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37217/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36335
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:19:25.753449 12693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:25.753773 12693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:25.753844 12693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:25.755304 12693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:25.755395 12693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:25.755471 12693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:19:25.757323 12693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:37217/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:25.757752 12693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:25.758065 12693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:25.764146 12699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:25.764218 12701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:25.764463 12698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:25.764575 12693 server_base.cc:1034] running on GCE node
I20260925 12:19:25.764909 12693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:25.765163 12693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:25.766369 12693 hybrid_clock.cc:648] HybridClock initialized: now 1790338765766347 us; error 35 us; skew 500 ppm
Sep 25 12:19:25 dist-test-slave-clmx krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:25.769613 12693 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:19:25.770093 12693 webserver.cc:460] Webserver started at http://127.0.98.1:36515/ using document root <none> and password file <none>
I20260925 12:19:25.770365 12693 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:25.770413 12693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:25.770545 12693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 12:19:25 dist-test-slave-clmx krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:19:25.784229 12693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "079d3df727204008aaf040ec419da761"
format_stamp: "Formatted at 2026-09-25 12:19:25 on dist-test-slave-clmx"
server_key: "def52fcfed456fd714a0c7139577b5b0"
server_key_iv: "42d4d21607c1ae5f064e37f780da969a"
server_key_version: "kuduclusterkey@0"
I20260925 12:19:25.784740 12693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "079d3df727204008aaf040ec419da761"
format_stamp: "Formatted at 2026-09-25 12:19:25 on dist-test-slave-clmx"
server_key: "def52fcfed456fd714a0c7139577b5b0"
server_key_iv: "42d4d21607c1ae5f064e37f780da969a"
server_key_version: "kuduclusterkey@0"
I20260925 12:19:25.786551 12693 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 12:19:25.797808 12708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:25.798014 12693 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 12:19:25.798099 12693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "079d3df727204008aaf040ec419da761"
format_stamp: "Formatted at 2026-09-25 12:19:25 on dist-test-slave-clmx"
server_key: "def52fcfed456fd714a0c7139577b5b0"
server_key_iv: "42d4d21607c1ae5f064e37f780da969a"
server_key_version: "kuduclusterkey@0"
I20260925 12:19:25.798171 12693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:25.817582 12693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:25.819749 12693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:25.819900 12693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:25.820114 12693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:19:25.820423 12693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:19:25.820504 12693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:25.820590 12693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:19:25.820655 12693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:25.826573 12693 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40473
I20260925 12:19:25.826992 12693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:19:25.827358 12823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40473 every 8 connection(s)
I20260925 12:19:25.830353 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 12693
I20260925 12:19:25.830441 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:19:25.830647 392 external_mini_cluster.cc:1428] Setting key f4df05e5c76f45fd3e8aed39bf5d9f9a
Sep 25 12:19:25 dist-test-slave-clmx krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:25.839690 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:25.828064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36335 (local address 127.0.98.1:48345)
0925 12:19:25.828289 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:25.828333 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:25.828956 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:25.829317 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:25.829321 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:25.829515 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:25.829735 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:25.829740 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:25.830657 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:25.830658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:25.833911 (+ 3253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:25.833913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:25.834393 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:25.834394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:25.834574 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:25.835035 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:25.835044 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:25.836398 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:25.838243 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:25.838248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:25.838250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:25.838514 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:25.838699 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:25.838699 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:25.838699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:25.838803 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:25.839067 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:25.839072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:25.839221 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 12:19:25.839317 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 12:19:25.839449 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
I20260925 12:19:25.840353 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:25.828103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48345 (local address 127.0.98.62:36335)
0925 12:19:25.829115 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:25.829117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:25.829124 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:19:25.829147 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:25.829148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:25.829160 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:25.829221 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:25.829837 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:25.830557 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:25.834001 (+ 3444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:25.834269 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:25.834324 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:25.836501 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:25.836502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:25.836502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:25.836509 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:25.838156 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:25.838589 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:25.838590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:25.838590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:25.838643 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:25.838860 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:25.838861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:25.838861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:25.838988 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:25.839044 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:25.840145 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0925 12:19:25.840234 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":987,"thread_start_us":78,"threads_started":1}
I20260925 12:19:25.841055 12824 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36335
I20260925 12:19:25.841135 12824 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:25.841307 12824 heartbeater.cc:511] Master 127.0.98.62:36335 requested a full tablet report, sending...
I20260925 12:19:25.841861 12612 ts_manager.cc:194] Registered new tserver with Master: 079d3df727204008aaf040ec419da761 (127.0.98.1:40473)
I20260925 12:19:25.842702 12612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48345
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:19:25.870478 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37217/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36335
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:19:25.983796 12830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:25.984113 12830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:25.984185 12830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:25.985852 12830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:25.985944 12830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:25.986019 12830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:19:25.987939 12830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:37217/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:25.988324 12830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:25.988579 12830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:25.994321 12838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:25.994671 12837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:25.995445 12830 server_base.cc:1034] running on GCE node
W20260925 12:19:25.995679 12840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:25.995915 12830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:25.996129 12830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:25.997262 12830 hybrid_clock.cc:648] HybridClock initialized: now 1790338765997228 us; error 47 us; skew 500 ppm
Sep 25 12:19:25 dist-test-slave-clmx krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:26.000294 12830 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:19:26.000710 12830 webserver.cc:460] Webserver started at http://127.0.98.2:34323/ using document root <none> and password file <none>
I20260925 12:19:26.000913 12830 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:26.000957 12830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:26.001060 12830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 12:19:26 dist-test-slave-clmx krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 12:19:26 dist-test-slave-clmx krb5kdc[12136](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790338766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:26.014451 12830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "c6d96969615f44b58c3780394b1d62ef"
format_stamp: "Formatted at 2026-09-25 12:19:26 on dist-test-slave-clmx"
server_key: "26b88614b07eb8db85e6d6186912928b"
server_key_iv: "bce7455cf3b055dfa51689b867dae6de"
server_key_version: "kuduclusterkey@0"
I20260925 12:19:26.015302 12830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c6d96969615f44b58c3780394b1d62ef"
format_stamp: "Formatted at 2026-09-25 12:19:26 on dist-test-slave-clmx"
server_key: "26b88614b07eb8db85e6d6186912928b"
server_key_iv: "bce7455cf3b055dfa51689b867dae6de"
server_key_version: "kuduclusterkey@0"
I20260925 12:19:26.016980 12830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:19:26.031414 12847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:26.031610 12830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 12:19:26.031706 12830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "c6d96969615f44b58c3780394b1d62ef"
format_stamp: "Formatted at 2026-09-25 12:19:26 on dist-test-slave-clmx"
server_key: "26b88614b07eb8db85e6d6186912928b"
server_key_iv: "bce7455cf3b055dfa51689b867dae6de"
server_key_version: "kuduclusterkey@0"
I20260925 12:19:26.031775 12830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:26.055655 12830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:26.057976 12830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:26.058146 12830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:26.058403 12830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:19:26.058758 12830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:19:26.058831 12830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:26.058897 12830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:19:26.058951 12830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:26.064812 12830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41077
I20260925 12:19:26.065222 12830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:19:26.068718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 12830
I20260925 12:19:26.068815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:19:26.069000 392 external_mini_cluster.cc:1428] Setting key 0c92ac3e9a5492f1afccfc324338b8a1
I20260925 12:19:26.076009 12960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41077 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 25 12:19:26 dist-test-slave-clmx krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:26.094106 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:26.076794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46747 (local address 127.0.98.62:36335)
0925 12:19:26.076851 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:26.076854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:26.081317 (+ 4463us) server_negotiation.cc:407] Connection header received
0925 12:19:26.082117 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:26.082119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:26.082131 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:26.082162 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:26.083038 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:26.083931 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:26.087527 (+ 3596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:26.087885 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:26.087931 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:26.090592 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:26.090594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:26.090594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:26.090602 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:26.092483 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:26.092970 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:26.092971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:26.092972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:26.093021 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:26.093287 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:26.093288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:26.093288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:26.093425 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:26.093471 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:26.093902 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 12:19:26.093990 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 12:19:26.094818 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:26.076562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36335 (local address 127.0.98.2:46747)
0925 12:19:26.081167 (+ 4605us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:26.081214 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:26.081858 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:26.082351 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:26.082355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:26.082594 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:26.082857 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:26.082864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:26.086151 (+ 3287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:26.086152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:26.087415 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:26.087419 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:26.088044 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:26.088045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:26.088183 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:26.088820 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:26.088829 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:26.090443 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:26.092588 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:26.092594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:26.092596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:26.092880 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:26.093089 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:26.093090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:26.093090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:26.093212 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:26.093533 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:26.093538 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:26.093715 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 12:19:26.094465 (+ 750us) client_negotiation.cc:240] Negotiation successful
0925 12:19:26.094642 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4561,"thread_start_us":74,"threads_started":1}
I20260925 12:19:26.095321 12961 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36335
I20260925 12:19:26.095433 12961 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:26.095681 12961 heartbeater.cc:511] Master 127.0.98.62:36335 requested a full tablet report, sending...
I20260925 12:19:26.096093 12612 ts_manager.cc:194] Registered new tserver with Master: c6d96969615f44b58c3780394b1d62ef (127.0.98.2:41077)
I20260925 12:19:26.096593 12612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:19:26.101976 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37217/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36335
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:19:26.210669 12966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:26.211009 12966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:26.211090 12966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:26.212772 12966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:26.212867 12966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:26.212944 12966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:19:26.214982 12966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:37217/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:26.215476 12966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:26.215772 12966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:26.221585 12972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:26.221859 12975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:26.222079 12973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:26.222170 12966 server_base.cc:1034] running on GCE node
I20260925 12:19:26.222452 12966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:26.222723 12966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:26.223893 12966 hybrid_clock.cc:648] HybridClock initialized: now 1790338766223856 us; error 45 us; skew 500 ppm
Sep 25 12:19:26 dist-test-slave-clmx krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:26.227016 12966 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:19:26.227490 12966 webserver.cc:460] Webserver started at http://127.0.98.3:36287/ using document root <none> and password file <none>
I20260925 12:19:26.227797 12966 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:26.227890 12966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:26.228048 12966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 12:19:26 dist-test-slave-clmx krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:19:26.248594 12966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "d4cd76b08ed940a38ecc13ef540f615b"
format_stamp: "Formatted at 2026-09-25 12:19:26 on dist-test-slave-clmx"
server_key: "7be63198a598cf6a37871aed170d56d1"
server_key_iv: "7e26c27391508ff3aee37b0c232c9280"
server_key_version: "kuduclusterkey@0"
I20260925 12:19:26.249099 12966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d4cd76b08ed940a38ecc13ef540f615b"
format_stamp: "Formatted at 2026-09-25 12:19:26 on dist-test-slave-clmx"
server_key: "7be63198a598cf6a37871aed170d56d1"
server_key_iv: "7e26c27391508ff3aee37b0c232c9280"
server_key_version: "kuduclusterkey@0"
I20260925 12:19:26.251449 12966 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 12:19:26.269145 12982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:26.269361 12966 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 12:19:26.269444 12966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "d4cd76b08ed940a38ecc13ef540f615b"
format_stamp: "Formatted at 2026-09-25 12:19:26 on dist-test-slave-clmx"
server_key: "7be63198a598cf6a37871aed170d56d1"
server_key_iv: "7e26c27391508ff3aee37b0c232c9280"
server_key_version: "kuduclusterkey@0"
I20260925 12:19:26.269532 12966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:26.280510 12966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:26.282675 12966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:26.282802 12966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:26.283010 12966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:19:26.283408 12966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:19:26.283455 12966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:26.283485 12966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:19:26.283552 12966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:26.290942 12966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38251
I20260925 12:19:26.291009 13095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38251 every 8 connection(s)
I20260925 12:19:26.291370 12966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:19:26.297906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 12966
I20260925 12:19:26.298026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:19:26.298184 392 external_mini_cluster.cc:1428] Setting key 51cc1bb28fb2e5401dad30c73d277cfb
Sep 25 12:19:26 dist-test-slave-clmx krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:26.302210 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:26.292092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49863 (local address 127.0.98.62:36335)
0925 12:19:26.292155 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:26.292157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:26.292339 (+ 182us) server_negotiation.cc:407] Connection header received
0925 12:19:26.292814 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:26.292816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:26.292827 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:26.292897 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:26.293363 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:26.294157 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:26.295487 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:26.295850 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:26.295937 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:26.298727 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:26.298728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:26.298729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:26.298736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:26.300477 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:26.300858 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:26.300859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:26.300860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:26.300932 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:26.301154 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:26.301155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:26.301156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:26.301729 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:26.301810 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:26.302054 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 12:19:26.302127 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 12:19:26.302372 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:26.292023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36335 (local address 127.0.98.3:49863)
0925 12:19:26.292216 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:26.292257 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:26.292729 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:26.292957 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:26.292960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:26.293112 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:26.293279 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:26.293282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:26.294656 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:26.294656 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:26.295395 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:26.295397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:26.295947 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:26.295948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:26.296040 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:26.296995 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:26.297002 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:26.298647 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:26.300575 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:26.300580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:26.300581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:26.300782 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:26.301001 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:26.301002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:26.301002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:26.301090 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:26.301801 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:26.301805 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:26.301927 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 12:19:26.302007 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 12:19:26.302145 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260925 12:19:26.302883 13096 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36335
I20260925 12:19:26.302982 13096 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:26.303205 13096 heartbeater.cc:511] Master 127.0.98.62:36335 requested a full tablet report, sending...
I20260925 12:19:26.303669 12612 ts_manager.cc:194] Registered new tserver with Master: d4cd76b08ed940a38ecc13ef540f615b (127.0.98.3:38251)
I20260925 12:19:26.304129 12612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49863
I20260925 12:19:26.313287 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:19:26.313933 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 12693
I20260925 12:19:26.320895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 12830
I20260925 12:19:26.326431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 12966
I20260925 12:19:26.330828 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 12580
I20260925 12:19:26.335434 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36335
--webserver_interface=127.0.98.62
--webserver_port=44181
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36335
--ranger_config_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37217/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37217/kms with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:19:26.412549 13099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:26.412719 13099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:26.412739 13099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:26.413681 13099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:19:26.413712 13099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:26.413723 13099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:26.413734 13099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:19:26.413745 13099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:19:26.415100 13099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37217/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36335
--ranger_config_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:26.415252 13099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:26.415390 13099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:26.419709 13105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:26.419723 13104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:26.419723 13107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:26.420087 13099 server_base.cc:1034] running on GCE node
I20260925 12:19:26.420290 13099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:26.420554 13099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:26.421715 13099 hybrid_clock.cc:648] HybridClock initialized: now 1790338766421705 us; error 25 us; skew 500 ppm
Sep 25 12:19:26 dist-test-slave-clmx krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:26.424067 13099 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:19:26.424501 13099 webserver.cc:460] Webserver started at http://127.0.98.62:44181/ using document root <none> and password file <none>
I20260925 12:19:26.424666 13099 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:26.424708 13099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 12:19:26 dist-test-slave-clmx krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:19:26.470587 13099 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260925 12:19:26.471326 13115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:26.471499 13099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:19:26.471601 13099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "912b6b185195439ab861aa3091f7ec9d"
format_stamp: "Formatted at 2026-09-25 12:19:24 on dist-test-slave-clmx"
server_key: "e1fabc7669fabc491007f5a00b792530"
server_key_iv: "385ebcb7a45fe6bee1bf8a3861581261"
server_key_version: "kuduclusterkey@0"
I20260925 12:19:26.471979 13099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:26.479905 13099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:26.481630 13099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:26.481763 13099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:26.486123 13099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36335
I20260925 12:19:26.486158 13167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36335 every 8 connection(s)
I20260925 12:19:26.486490 13099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:19:26.487953 13168 sys_catalog.cc:263] Verifying existing consensus state
I20260925 12:19:26.488746 13168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d: Bootstrap starting.
I20260925 12:19:26.490110 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 13099
I20260925 12:19:26.490384 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37217/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40473
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36335
--webserver_port=36515
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37217/kms with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:19:26.498718 13168 log.cc:824] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:26.500836 13168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:19:26.501145 13168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d: Bootstrap complete.
I20260925 12:19:26.505221 13168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912b6b185195439ab861aa3091f7ec9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36335 } }
I20260925 12:19:26.505537 13168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912b6b185195439ab861aa3091f7ec9d, State: Initialized, Role: FOLLOWER
I20260925 12:19:26.505745 13168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912b6b185195439ab861aa3091f7ec9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36335 } }
I20260925 12:19:26.505810 13168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:19:26.505844 13168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:19:26.505887 13168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [term 1 FOLLOWER]: Advancing to term 2
I20260925 12:19:26.507057 13168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912b6b185195439ab861aa3091f7ec9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36335 } }
I20260925 12:19:26.507158 13168 leader_election.cc:302] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 912b6b185195439ab861aa3091f7ec9d; no voters:
I20260925 12:19:26.507289 13168 leader_election.cc:288] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 12:19:26.507362 13172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [term 2 FOLLOWER]: Leader election won for term 2
I20260925 12:19:26.507506 13172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [term 2 LEADER]: Becoming Leader. State: Replica: 912b6b185195439ab861aa3091f7ec9d, State: Running, Role: LEADER
I20260925 12:19:26.507509 13168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:19:26.507601 13172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912b6b185195439ab861aa3091f7ec9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36335 } }
I20260925 12:19:26.507865 13173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "912b6b185195439ab861aa3091f7ec9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912b6b185195439ab861aa3091f7ec9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36335 } } }
I20260925 12:19:26.507942 13173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [sys.catalog]: This master's current role is: LEADER
I20260925 12:19:26.508093 13173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 912b6b185195439ab861aa3091f7ec9d. Latest consensus state: current_term: 2 leader_uuid: "912b6b185195439ab861aa3091f7ec9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912b6b185195439ab861aa3091f7ec9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36335 } } }
I20260925 12:19:26.508157 13173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d [sys.catalog]: This master's current role is: LEADER
I20260925 12:19:26.508447 13179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:19:26.508896 13179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:19:26.509034 13179 catalog_manager.cc:1259] Loaded cluster ID: 62bb019056ee498e8aa2aa1bf7556073
I20260925 12:19:26.509066 13179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:19:26.509179 13168 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 12:19:26.512348 13179 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:19:26.512915 13179 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d: Loaded TSK: 0
I20260925 12:19:26.513180 13179 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 12:19:26.603588 13170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:26.603859 13170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:26.603893 13170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:26.605401 13170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:26.605453 13170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:26.605491 13170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:19:26.607383 13170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37217/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36515
--tserver_master_addrs=127.0.98.62:36335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:26.607573 13170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:26.607801 13170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:26.613763 13209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:26.614046 13207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:26.614543 13170 server_base.cc:1034] running on GCE node
W20260925 12:19:26.614635 13206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:26.614852 13170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:26.615109 13170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:26.616282 13170 hybrid_clock.cc:648] HybridClock initialized: now 1790338766616264 us; error 30 us; skew 500 ppm
Sep 25 12:19:26 dist-test-slave-clmx krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:26.619362 13170 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:19:26.619891 13170 webserver.cc:460] Webserver started at http://127.0.98.1:36515/ using document root <none> and password file <none>
I20260925 12:19:26.620124 13170 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:26.620199 13170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 12:19:26 dist-test-slave-clmx krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:19:26.643968 13170 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260925 12:19:26.650460 13217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:26.650849 13170 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260925 12:19:26.650926 13170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "079d3df727204008aaf040ec419da761"
format_stamp: "Formatted at 2026-09-25 12:19:25 on dist-test-slave-clmx"
server_key: "def52fcfed456fd714a0c7139577b5b0"
server_key_iv: "42d4d21607c1ae5f064e37f780da969a"
server_key_version: "kuduclusterkey@0"
I20260925 12:19:26.651361 13170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:26.692885 13170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:26.695004 13170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:26.695178 13170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:26.695381 13170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:19:26.695765 13170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:19:26.695809 13170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:26.695839 13170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:19:26.695863 13170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:26.701395 13170 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40473
I20260925 12:19:26.701843 13170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:19:26.703851 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 13170
I20260925 12:19:26.704145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37217/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41077
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36335
--webserver_port=34323
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37217/kms with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:19:26.706804 13330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40473 every 8 connection(s)
Sep 25 12:19:26 dist-test-slave-clmx krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:26.718475 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:26.707031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36335 (local address 127.0.98.1:33501)
0925 12:19:26.707443 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:26.707486 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:26.708263 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:26.708876 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:26.708881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:26.709137 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:26.709305 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:26.709310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:26.710980 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:26.710981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:26.711829 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:26.711831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:26.712418 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:26.712419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:26.712554 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:26.713034 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:26.713042 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:26.714519 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:26.716782 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:26.716787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:26.716789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:26.717042 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:26.717297 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:26.717297 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:26.717298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:26.717412 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:26.717873 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:26.717878 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:26.718024 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 12:19:26.718124 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 12:19:26.718255 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":85,"threads_started":1}
I20260925 12:19:26.719151 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:26.707223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33501 (local address 127.0.98.62:36335)
0925 12:19:26.707444 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:26.707453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:26.707639 (+ 186us) server_negotiation.cc:407] Connection header received
0925 12:19:26.708512 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:26.708519 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:26.708574 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:26.708710 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:26.709417 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:26.710883 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:26.711927 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:26.712317 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:26.712435 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:26.714638 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:26.714642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:26.714642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:26.714653 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:26.716690 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:26.717129 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:26.717131 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:26.717132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:26.717225 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:26.717493 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:26.717494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:26.717494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:26.717771 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:26.717846 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:26.718874 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0925 12:19:26.718979 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
I20260925 12:19:26.719806 13331 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36335
I20260925 12:19:26.719883 13331 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:26.720033 13331 heartbeater.cc:511] Master 127.0.98.62:36335 requested a full tablet report, sending...
W20260925 12:19:26.720842 13331 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36335 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 12:19:26.720901 13331 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:26.720953 13331 heartbeater.cc:511] Master 127.0.98.62:36335 requested a full tablet report, sending...
I20260925 12:19:26.721267 13331 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:26.721369 13331 heartbeater.cc:511] Master 127.0.98.62:36335 requested a full tablet report, sending...
W20260925 12:19:26.721688 13331 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 12:19:26.812399 13333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:26.812674 13333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:26.812709 13333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:26.813779 13333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:26.813835 13333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:26.813874 13333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:19:26.815449 13333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37217/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34323
--tserver_master_addrs=127.0.98.62:36335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:26.815673 13333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:26.815857 13333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:26.820477 13340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:26.820499 13341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:26.820678 13343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:26.820881 13333 server_base.cc:1034] running on GCE node
I20260925 12:19:26.821055 13333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:26.821285 13333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:26.822460 13333 hybrid_clock.cc:648] HybridClock initialized: now 1790338766822475 us; error 53 us; skew 500 ppm
Sep 25 12:19:26 dist-test-slave-clmx krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:26.825068 13333 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:19:26.825511 13333 webserver.cc:460] Webserver started at http://127.0.98.2:34323/ using document root <none> and password file <none>
I20260925 12:19:26.825723 13333 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:26.825798 13333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 12:19:26 dist-test-slave-clmx krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:19:26.840607 13333 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 12:19:26.841457 13351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:26.841670 13333 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 12:19:26.841758 13333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "c6d96969615f44b58c3780394b1d62ef"
format_stamp: "Formatted at 2026-09-25 12:19:26 on dist-test-slave-clmx"
server_key: "26b88614b07eb8db85e6d6186912928b"
server_key_iv: "bce7455cf3b055dfa51689b867dae6de"
server_key_version: "kuduclusterkey@0"
I20260925 12:19:26.842074 13333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:26.858261 13333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:26.860308 13333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:26.860440 13333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:26.860670 13333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:19:26.861037 13333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:19:26.861090 13333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:26.861126 13333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:19:26.861151 13333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:26.870086 13333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41077
I20260925 12:19:26.870448 13333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:19:26.872107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 13333
I20260925 12:19:26.872437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37217/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38251
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36335
--webserver_port=36287
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37217/kms with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:19:26.875765 13464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41077 every 8 connection(s)
Sep 25 12:19:26 dist-test-slave-clmx krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:26.896834 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:26.876360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38523 (local address 127.0.98.62:36335)
0925 12:19:26.876410 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:26.876412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:26.884781 (+ 8369us) server_negotiation.cc:407] Connection header received
0925 12:19:26.885475 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:26.885476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:26.885489 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:26.885521 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:26.886346 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:26.887105 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:26.888635 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:26.888914 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:26.888949 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:26.891610 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:26.891631 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:26.891632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:26.891639 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:26.893231 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:26.895796 (+ 2565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:26.895797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:26.895798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:26.895857 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:26.896119 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:26.896120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:26.896120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:26.896254 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:26.896302 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:26.896664 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 12:19:26.896745 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:19:26.897259 13468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:26.876379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36335 (local address 127.0.98.2:38523)
0925 12:19:26.884622 (+ 8243us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:26.884671 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:26.885348 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:26.885760 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:26.885764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:26.885986 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:26.886244 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:26.886248 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:26.887223 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:26.887224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:26.888539 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:26.888542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:26.889036 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:26.889037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:26.889154 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:26.889755 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:26.889765 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:26.891502 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:26.893322 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:26.893327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:26.893330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:26.893596 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:26.895929 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:26.895930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:26.895931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:26.896053 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:26.896330 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:26.896335 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:26.896522 (+ 187us) client_negotiation.cc:769] Sending connection context
0925 12:19:26.896930 (+ 408us) client_negotiation.cc:240] Negotiation successful
0925 12:19:26.897068 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8192,"thread_start_us":94,"threads_started":1}
I20260925 12:19:26.897688 13465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36335
I20260925 12:19:26.897787 13465 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:26.897967 13465 heartbeater.cc:511] Master 127.0.98.62:36335 requested a full tablet report, sending...
W20260925 12:19:26.898494 13465 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36335 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 12:19:26.898571 13465 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:26.898641 13465 heartbeater.cc:511] Master 127.0.98.62:36335 requested a full tablet report, sending...
I20260925 12:19:26.898891 13465 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:26.898965 13465 heartbeater.cc:511] Master 127.0.98.62:36335 requested a full tablet report, sending...
W20260925 12:19:26.899222 13465 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 12:19:26.982281 13467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:26.982528 13467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:26.982561 13467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:26.984405 13467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:26.984467 13467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:26.984511 13467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:19:26.986536 13467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37217/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36287
--tserver_master_addrs=127.0.98.62:36335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:26.986764 13467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:26.986997 13467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:26.993739 13474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:26.993728 13477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:26.993821 13475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:26.994238 13467 server_base.cc:1034] running on GCE node
I20260925 12:19:26.994395 13467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:26.994623 13467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:26.995771 13467 hybrid_clock.cc:648] HybridClock initialized: now 1790338766995741 us; error 44 us; skew 500 ppm
Sep 25 12:19:26 dist-test-slave-clmx krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:26.998661 13467 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:19:26.999138 13467 webserver.cc:460] Webserver started at http://127.0.98.3:36287/ using document root <none> and password file <none>
I20260925 12:19:26.999388 13467 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:26.999454 13467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 12:19:26 dist-test-slave-clmx krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:19:27.020085 13467 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.000s
I20260925 12:19:27.020897 13485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:27.021090 13467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:19:27.021179 13467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "d4cd76b08ed940a38ecc13ef540f615b"
format_stamp: "Formatted at 2026-09-25 12:19:26 on dist-test-slave-clmx"
server_key: "7be63198a598cf6a37871aed170d56d1"
server_key_iv: "7e26c27391508ff3aee37b0c232c9280"
server_key_version: "kuduclusterkey@0"
I20260925 12:19:27.021592 13467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:27.046742 13467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:27.048799 13467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:27.048946 13467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:27.049176 13467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:19:27.049484 13467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:19:27.049532 13467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:27.049571 13467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:19:27.049594 13467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:27.055874 13467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38251
I20260925 12:19:27.056299 13467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:19:27.057636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 13467
I20260925 12:19:27.058190 13598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38251 every 8 connection(s)
Sep 25 12:19:27 dist-test-slave-clmx krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:27.068498 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:27.058514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36335 (local address 127.0.98.3:39755)
0925 12:19:27.059001 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:27.059044 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:27.059485 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:27.059746 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:27.059751 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:27.059961 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:27.060195 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:27.060201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:27.061131 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:27.061132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:27.062357 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:27.062360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:27.062895 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:27.062896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:27.063035 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:27.063583 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:27.063592 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:27.065049 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:27.066849 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:27.066855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:27.066857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:27.067131 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:27.067403 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:27.067404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:27.067404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:27.067541 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:27.067867 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:27.067873 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:27.068041 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 12:19:27.068152 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 12:19:27.068290 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":110,"threads_started":1}
I20260925 12:19:27.068740 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:27.058078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40504 (local address 127.0.98.62:36335)
0925 12:19:27.058236 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:27.058238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:27.058874 (+ 636us) server_negotiation.cc:407] Connection header received
0925 12:19:27.058951 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:27.058951 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:27.058971 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:27.059028 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:27.059759 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:27.060444 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:27.061173 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:27.061397 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:27.061469 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:27.063695 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:27.063696 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:27.063697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:27.063704 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:27.065646 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:27.067869 (+ 2223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:27.067870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:27.067871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:27.067927 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:27.068122 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:27.068123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:27.068123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:27.068230 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:27.068358 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:27.068578 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 12:19:27.068653 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260925 12:19:27.069154 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:27.058500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39755 (local address 127.0.98.62:36335)
0925 12:19:27.058631 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:27.058633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:27.059104 (+ 471us) server_negotiation.cc:407] Connection header received
0925 12:19:27.059578 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:27.059579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:27.059590 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:27.059668 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:27.060305 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:27.061024 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:27.062456 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:27.062759 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:27.062822 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:27.065151 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:27.065152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:27.065152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:27.065159 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:27.066759 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:27.067238 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:27.067240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:27.067240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:27.067314 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:27.067618 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:27.067619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:27.067619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:27.067778 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:27.067840 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:27.068980 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0925 12:19:27.069069 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260925 12:19:27.069846 13599 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36335
I20260925 12:19:27.069924 13599 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:27.070086 13599 heartbeater.cc:511] Master 127.0.98.62:36335 requested a full tablet report, sending...
W20260925 12:19:27.070773 13599 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36335 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 12:19:27.070830 13599 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:27.070879 13599 heartbeater.cc:511] Master 127.0.98.62:36335 requested a full tablet report, sending...
I20260925 12:19:27.071308 13599 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:27.071399 13599 heartbeater.cc:511] Master 127.0.98.62:36335 requested a full tablet report, sending...
W20260925 12:19:27.071831 13599 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 12:19:27 dist-test-slave-clmx krb5kdc[12136](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790338767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:27.721839 13331 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:27.722000 13331 heartbeater.cc:511] Master 127.0.98.62:36335 requested a full tablet report, sending...
I20260925 12:19:27.722319 13132 ts_manager.cc:194] Registered new tserver with Master: 079d3df727204008aaf040ec419da761 (127.0.98.1:40473)
I20260925 12:19:27.723021 13132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33501
Sep 25 12:19:27 dist-test-slave-clmx krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:19:27.899401 13465 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:27.899533 13465 heartbeater.cc:511] Master 127.0.98.62:36335 requested a full tablet report, sending...
I20260925 12:19:27.899853 13132 ts_manager.cc:194] Registered new tserver with Master: c6d96969615f44b58c3780394b1d62ef (127.0.98.2:41077)
I20260925 12:19:27.900343 13132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38523
I20260925 12:19:28.072000 13599 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:28.072134 13599 heartbeater.cc:511] Master 127.0.98.62:36335 requested a full tablet report, sending...
I20260925 12:19:28.072394 13132 ts_manager.cc:194] Registered new tserver with Master: d4cd76b08ed940a38ecc13ef540f615b (127.0.98.3:38251)
I20260925 12:19:28.072916 13132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39755
Sep 25 12:19:28 dist-test-slave-clmx krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:19:28.085137 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:19:28.103253 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:28.087019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40516 (local address 127.0.98.62:36335)
0925 12:19:28.087154 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:28.087156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:28.089168 (+ 2012us) server_negotiation.cc:407] Connection header received
0925 12:19:28.089242 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:28.089244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:28.089283 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:28.089563 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:28.091205 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.094037 (+ 2832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.096646 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.097598 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.097826 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:28.099794 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:28.099795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:28.099795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:28.099801 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:28.101733 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:28.102091 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:28.102092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:28.102093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:28.102145 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:28.102310 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:28.102310 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:28.102310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:28.102621 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:28.102703 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:28.103041 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 12:19:28.103128 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260925 12:19:28.106293 13132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:19:28.108579 13132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:19:28.126189 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:28.120501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41077 (local address 127.0.0.1:60558)
0925 12:19:28.120917 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:28.121116 (+ 199us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:28.121185 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:28.121778 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:28.121782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:28.121784 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:19:28.121851 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:28.121853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.123080 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.123081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:28.124730 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:28.124733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.125909 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.125910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:28.125985 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:28.126000 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 12:19:28.126056 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 12:19:28.126103 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":79,"threads_started":1}
I20260925 12:19:28.126493 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:28.120754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.0.98.2:41077)
0925 12:19:28.121074 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:28.121079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:28.121269 (+ 190us) server_negotiation.cc:407] Connection header received
0925 12:19:28.121307 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:28.121309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:28.121324 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:28.121407 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:19:28.121943 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.122987 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.124823 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.125532 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.125599 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:28.126348 (+ 749us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:28.126376 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 12:19:28.126425 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":60,"threads_started":1}
I20260925 12:19:28.127013 13399 tablet_service.cc:1467] Processing CreateTablet for tablet 0f149c8a30474092ad81b6b811db83e7 (DEFAULT_TABLE table=test-table [id=61ba012acffa48978a57c7056101a422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:19:28.127326 13399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f149c8a30474092ad81b6b811db83e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:19:28.132314 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:28.120501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40473 (local address 127.0.0.1:56092)
0925 12:19:28.123106 (+ 2605us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:28.123129 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:28.123189 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:28.128766 (+ 5577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:28.128767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:28.128769 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:19:28.128802 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:28.128804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.129967 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.129968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:28.130957 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:28.130959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.132107 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.132108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:28.132166 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:28.132171 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:19:28.132203 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 12:19:28.132240 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2591,"mutex_wait_us":3,"thread_start_us":77,"threads_started":1}
I20260925 12:19:28.132916 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:28.120633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56092 (local address 127.0.98.1:40473)
0925 12:19:28.128282 (+ 7649us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:28.128285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:28.128295 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:19:28.128326 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:28.128328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:28.128342 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:28.128691 (+ 349us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:19:28.128874 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.129868 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.131052 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.132017 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.132689 (+ 672us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:28.132769 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:28.132795 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 12:19:28.132831 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7614,"thread_start_us":58,"threads_started":1}
I20260925 12:19:28.133194 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:28.120912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59286 (local address 127.0.98.3:38251)
0925 12:19:28.123291 (+ 2379us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:28.123295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:28.123439 (+ 144us) server_negotiation.cc:407] Connection header received
0925 12:19:28.123473 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:28.123475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:28.123491 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:28.123560 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:19:28.123769 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.124761 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.126029 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.128077 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.128149 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:28.128204 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:28.133075 (+ 4871us) server_negotiation.cc:299] Negotiation successful
0925 12:19:28.133122 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2347,"thread_start_us":60,"threads_started":1}
I20260925 12:19:28.133399 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:28.120725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38251 (local address 127.0.0.1:59286)
0925 12:19:28.123337 (+ 2612us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:28.123346 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:28.123387 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:28.123656 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:28.123657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:28.123659 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:19:28.123695 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:28.123697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.124866 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.124866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:28.125953 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:28.125955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.132220 (+ 6265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.132221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:28.132271 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:28.132979 (+ 708us) client_negotiation.cc:769] Sending connection context
0925 12:19:28.133290 (+ 311us) client_negotiation.cc:240] Negotiation successful
0925 12:19:28.133338 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2598,"thread_start_us":57,"threads_started":1}
I20260925 12:19:28.133746 13635 tablet_bootstrap.cc:492] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef: Bootstrap starting.
I20260925 12:19:28.134377 13635 tablet_bootstrap.cc:654] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef: Neither blocks nor log segments found. Creating new log.
I20260925 12:19:28.134670 13635 log.cc:824] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:28.135097 13254 tablet_service.cc:1467] Processing CreateTablet for tablet 0f149c8a30474092ad81b6b811db83e7 (DEFAULT_TABLE table=test-table [id=61ba012acffa48978a57c7056101a422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:19:28.135406 13254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f149c8a30474092ad81b6b811db83e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:19:28.135756 13635 tablet_bootstrap.cc:492] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef: No bootstrap required, opened a new log
I20260925 12:19:28.135833 13635 ts_tablet_manager.cc:1397] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:19:28.137645 13635 raft_consensus.cc:348] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d96969615f44b58c3780394b1d62ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "d4cd76b08ed940a38ecc13ef540f615b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38251 } } peers { permanent_uuid: "079d3df727204008aaf040ec419da761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40473 } }
I20260925 12:19:28.137786 13635 raft_consensus.cc:374] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:19:28.137825 13635 raft_consensus.cc:729] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d96969615f44b58c3780394b1d62ef, State: Initialized, Role: FOLLOWER
I20260925 12:19:28.137924 13635 consensus_queue.cc:260] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d96969615f44b58c3780394b1d62ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "d4cd76b08ed940a38ecc13ef540f615b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38251 } } peers { permanent_uuid: "079d3df727204008aaf040ec419da761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40473 } }
I20260925 12:19:28.138188 13635 ts_tablet_manager.cc:1428] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:19:28.138195 13465 heartbeater.cc:503] Master 127.0.98.62:36335 was elected leader, sending a full tablet report...
I20260925 12:19:28.138959 13638 tablet_bootstrap.cc:492] T 0f149c8a30474092ad81b6b811db83e7 P 079d3df727204008aaf040ec419da761: Bootstrap starting.
I20260925 12:19:28.138929 13533 tablet_service.cc:1467] Processing CreateTablet for tablet 0f149c8a30474092ad81b6b811db83e7 (DEFAULT_TABLE table=test-table [id=61ba012acffa48978a57c7056101a422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:19:28.139251 13533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f149c8a30474092ad81b6b811db83e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:19:28.139588 13638 tablet_bootstrap.cc:654] T 0f149c8a30474092ad81b6b811db83e7 P 079d3df727204008aaf040ec419da761: Neither blocks nor log segments found. Creating new log.
I20260925 12:19:28.139909 13638 log.cc:824] T 0f149c8a30474092ad81b6b811db83e7 P 079d3df727204008aaf040ec419da761: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:28.140556 13638 tablet_bootstrap.cc:492] T 0f149c8a30474092ad81b6b811db83e7 P 079d3df727204008aaf040ec419da761: No bootstrap required, opened a new log
I20260925 12:19:28.140635 13638 ts_tablet_manager.cc:1397] T 0f149c8a30474092ad81b6b811db83e7 P 079d3df727204008aaf040ec419da761: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:19:28.142349 13640 tablet_bootstrap.cc:492] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b: Bootstrap starting.
I20260925 12:19:28.142638 13638 raft_consensus.cc:348] T 0f149c8a30474092ad81b6b811db83e7 P 079d3df727204008aaf040ec419da761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d96969615f44b58c3780394b1d62ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "d4cd76b08ed940a38ecc13ef540f615b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38251 } } peers { permanent_uuid: "079d3df727204008aaf040ec419da761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40473 } }
I20260925 12:19:28.142798 13638 raft_consensus.cc:374] T 0f149c8a30474092ad81b6b811db83e7 P 079d3df727204008aaf040ec419da761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:19:28.142840 13638 raft_consensus.cc:729] T 0f149c8a30474092ad81b6b811db83e7 P 079d3df727204008aaf040ec419da761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 079d3df727204008aaf040ec419da761, State: Initialized, Role: FOLLOWER
I20260925 12:19:28.142976 13638 consensus_queue.cc:260] T 0f149c8a30474092ad81b6b811db83e7 P 079d3df727204008aaf040ec419da761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d96969615f44b58c3780394b1d62ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "d4cd76b08ed940a38ecc13ef540f615b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38251 } } peers { permanent_uuid: "079d3df727204008aaf040ec419da761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40473 } }
I20260925 12:19:28.143179 13640 tablet_bootstrap.cc:654] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b: Neither blocks nor log segments found. Creating new log.
I20260925 12:19:28.143258 13638 ts_tablet_manager.cc:1428] T 0f149c8a30474092ad81b6b811db83e7 P 079d3df727204008aaf040ec419da761: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:19:28.143378 13331 heartbeater.cc:503] Master 127.0.98.62:36335 was elected leader, sending a full tablet report...
I20260925 12:19:28.143569 13640 log.cc:824] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:28.144332 13640 tablet_bootstrap.cc:492] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b: No bootstrap required, opened a new log
I20260925 12:19:28.144409 13640 ts_tablet_manager.cc:1397] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 12:19:28.146332 13640 raft_consensus.cc:348] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d96969615f44b58c3780394b1d62ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "d4cd76b08ed940a38ecc13ef540f615b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38251 } } peers { permanent_uuid: "079d3df727204008aaf040ec419da761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40473 } }
I20260925 12:19:28.146467 13640 raft_consensus.cc:374] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:19:28.146500 13640 raft_consensus.cc:729] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cd76b08ed940a38ecc13ef540f615b, State: Initialized, Role: FOLLOWER
I20260925 12:19:28.146595 13640 consensus_queue.cc:260] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d96969615f44b58c3780394b1d62ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "d4cd76b08ed940a38ecc13ef540f615b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38251 } } peers { permanent_uuid: "079d3df727204008aaf040ec419da761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40473 } }
I20260925 12:19:28.146818 13640 ts_tablet_manager.cc:1428] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 12:19:28.146946 13599 heartbeater.cc:503] Master 127.0.98.62:36335 was elected leader, sending a full tablet report...
W20260925 12:19:28.207553 13332 tablet.cc:2369] T 0f149c8a30474092ad81b6b811db83e7 P 079d3df727204008aaf040ec419da761: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:19:28.294797 13643 raft_consensus.cc:482] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:19:28.294945 13643 raft_consensus.cc:504] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d96969615f44b58c3780394b1d62ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "d4cd76b08ed940a38ecc13ef540f615b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38251 } } peers { permanent_uuid: "079d3df727204008aaf040ec419da761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40473 } }
I20260925 12:19:28.295348 13643 leader_election.cc:288] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6d96969615f44b58c3780394b1d62ef (127.0.98.2:41077), 079d3df727204008aaf040ec419da761 (127.0.98.1:40473)
I20260925 12:19:28.299167 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:28.295758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45299 (local address 127.0.98.2:41077)
0925 12:19:28.295800 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:28.295803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:28.296049 (+ 246us) server_negotiation.cc:407] Connection header received
0925 12:19:28.296080 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:28.296081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:28.296106 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:28.296148 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:19:28.296343 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.296959 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.298290 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.298814 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.298854 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:28.298864 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:28.299067 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 12:19:28.299103 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:19:28.299417 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:28.295526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41077 (local address 127.0.98.3:45299)
0925 12:19:28.295925 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:28.295936 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:28.296007 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:28.296197 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:28.296198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:28.296200 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:19:28.296283 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:28.296285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.297054 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.297055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:28.298196 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:28.298199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.298914 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.298915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:28.298972 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:28.298976 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:19:28.299277 (+ 301us) client_negotiation.cc:240] Negotiation successful
0925 12:19:28.299316 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":128,"threads_started":1}
I20260925 12:19:28.299814 13415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f149c8a30474092ad81b6b811db83e7" candidate_uuid: "d4cd76b08ed940a38ecc13ef540f615b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d96969615f44b58c3780394b1d62ef" is_pre_election: true
I20260925 12:19:28.299926 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:28.295790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55101 (local address 127.0.98.1:40473)
0925 12:19:28.295829 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:28.295831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:28.296132 (+ 301us) server_negotiation.cc:407] Connection header received
0925 12:19:28.296219 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:28.296220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:28.296232 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:28.296258 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:19:28.296476 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.297135 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.298491 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.299049 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.299095 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:28.299105 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:28.299827 (+ 722us) server_negotiation.cc:299] Negotiation successful
0925 12:19:28.299868 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:19:28.300009 13415 raft_consensus.cc:2454] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4cd76b08ed940a38ecc13ef540f615b in term 0.
I20260925 12:19:28.299925 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:28.295575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40473 (local address 127.0.98.3:55101)
0925 12:19:28.296072 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:28.296080 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:28.296152 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:28.296314 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:28.296314 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:28.296316 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:19:28.296375 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:28.296377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.297246 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.297247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:28.298371 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:28.298373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.299182 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.299183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:28.299749 (+ 566us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:28.299753 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:19:28.299813 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 12:19:28.299856 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":104,"threads_started":1}
I20260925 12:19:28.300237 13486 leader_election.cc:302] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6d96969615f44b58c3780394b1d62ef, d4cd76b08ed940a38ecc13ef540f615b; no voters:
I20260925 12:19:28.300491 13643 raft_consensus.cc:2781] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:19:28.300539 13643 raft_consensus.cc:482] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:19:28.300558 13643 raft_consensus.cc:3037] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:19:28.300496 13282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f149c8a30474092ad81b6b811db83e7" candidate_uuid: "d4cd76b08ed940a38ecc13ef540f615b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "079d3df727204008aaf040ec419da761" is_pre_election: true
I20260925 12:19:28.300653 13282 raft_consensus.cc:2454] T 0f149c8a30474092ad81b6b811db83e7 P 079d3df727204008aaf040ec419da761 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4cd76b08ed940a38ecc13ef540f615b in term 0.
I20260925 12:19:28.301450 13643 raft_consensus.cc:504] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d96969615f44b58c3780394b1d62ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "d4cd76b08ed940a38ecc13ef540f615b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38251 } } peers { permanent_uuid: "079d3df727204008aaf040ec419da761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40473 } }
I20260925 12:19:28.301579 13643 leader_election.cc:288] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [CANDIDATE]: Term 1 election: Requested vote from peers c6d96969615f44b58c3780394b1d62ef (127.0.98.2:41077), 079d3df727204008aaf040ec419da761 (127.0.98.1:40473)
I20260925 12:19:28.301743 13415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f149c8a30474092ad81b6b811db83e7" candidate_uuid: "d4cd76b08ed940a38ecc13ef540f615b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d96969615f44b58c3780394b1d62ef"
I20260925 12:19:28.301820 13415 raft_consensus.cc:3037] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:19:28.301885 13282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f149c8a30474092ad81b6b811db83e7" candidate_uuid: "d4cd76b08ed940a38ecc13ef540f615b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "079d3df727204008aaf040ec419da761"
I20260925 12:19:28.301955 13282 raft_consensus.cc:3037] T 0f149c8a30474092ad81b6b811db83e7 P 079d3df727204008aaf040ec419da761 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:19:28.302690 13415 raft_consensus.cc:2454] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4cd76b08ed940a38ecc13ef540f615b in term 1.
I20260925 12:19:28.302817 13282 raft_consensus.cc:2454] T 0f149c8a30474092ad81b6b811db83e7 P 079d3df727204008aaf040ec419da761 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4cd76b08ed940a38ecc13ef540f615b in term 1.
I20260925 12:19:28.302851 13486 leader_election.cc:302] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6d96969615f44b58c3780394b1d62ef, d4cd76b08ed940a38ecc13ef540f615b; no voters:
I20260925 12:19:28.303056 13643 raft_consensus.cc:2781] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:19:28.303256 13643 raft_consensus.cc:686] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [term 1 LEADER]: Becoming Leader. State: Replica: d4cd76b08ed940a38ecc13ef540f615b, State: Running, Role: LEADER
I20260925 12:19:28.303341 13643 consensus_queue.cc:237] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d96969615f44b58c3780394b1d62ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "d4cd76b08ed940a38ecc13ef540f615b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38251 } } peers { permanent_uuid: "079d3df727204008aaf040ec419da761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40473 } }
I20260925 12:19:28.304111 13132 catalog_manager.cc:5495] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b reported cstate change: term changed from 0 to 1, leader changed from <none> to d4cd76b08ed940a38ecc13ef540f615b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d4cd76b08ed940a38ecc13ef540f615b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d96969615f44b58c3780394b1d62ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4cd76b08ed940a38ecc13ef540f615b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "079d3df727204008aaf040ec419da761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40473 } health_report { overall_health: UNKNOWN } } }
W20260925 12:19:28.308667 13600 tablet.cc:2369] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:19:28.352577 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:28.349341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.0.98.3:38251)
0925 12:19:28.349387 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:28.349389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:28.349397 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:19:28.349451 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:28.349452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:28.349471 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:28.349562 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:19:28.349695 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.350399 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.351517 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:28.351826 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:28.351886 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:28.352086 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:19:28.352213 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:19:28.352286 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:28.352483 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 12:19:28.352505 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 12:19:28.354713 13282 raft_consensus.cc:1261] T 0f149c8a30474092ad81b6b811db83e7 P 079d3df727204008aaf040ec419da761 [term 1 FOLLOWER]: Refusing update from remote peer d4cd76b08ed940a38ecc13ef540f615b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:19:28.354812 13415 raft_consensus.cc:1261] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef [term 1 FOLLOWER]: Refusing update from remote peer d4cd76b08ed940a38ecc13ef540f615b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:19:28.355063 13643 consensus_queue.cc:1035] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [LEADER]: Connected to new peer: Peer: permanent_uuid: "079d3df727204008aaf040ec419da761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:19:28.355329 13643 consensus_queue.cc:1035] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d96969615f44b58c3780394b1d62ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:19:28.357901 13656 mvcc.cc:205] Tried to move back new op lower bound from 7333227595178270720 to 7333227594971009024. Current Snapshot: MvccSnapshot[applied={T|T < 7333227595178270720}]
I20260925 12:19:28.357983 13657 mvcc.cc:205] Tried to move back new op lower bound from 7333227595178270720 to 7333227594971009024. Current Snapshot: MvccSnapshot[applied={T|T < 7333227595178270720}]
I20260925 12:19:28.359551 13659 mvcc.cc:205] Tried to move back new op lower bound from 7333227595178270720 to 7333227594971009024. Current Snapshot: MvccSnapshot[applied={T|T < 7333227595178270720}]
I20260925 12:19:28.361183 13132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:19:28.361267 13132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:19:28.361991 13132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 912b6b185195439ab861aa3091f7ec9d: Sending DeleteTablet for 3 replicas of tablet 0f149c8a30474092ad81b6b811db83e7
I20260925 12:19:28.362298 13533 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f149c8a30474092ad81b6b811db83e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:19:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59286
I20260925 12:19:28.362710 13254 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f149c8a30474092ad81b6b811db83e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:19:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56092
I20260925 12:19:28.363215 13399 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f149c8a30474092ad81b6b811db83e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:19:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60558
I20260925 12:19:28.363546 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 13170
I20260925 12:19:28.365346 13665 tablet_replica.cc:330] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b: stopping tablet replica
I20260925 12:19:28.365471 13665 raft_consensus.cc:2229] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:19:28.365607 13665 raft_consensus.cc:2258] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:19:28.366096 13665 ts_tablet_manager.cc:1905] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:19:28.366231 13667 tablet_replica.cc:330] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef: stopping tablet replica
I20260925 12:19:28.366339 13667 raft_consensus.cc:2229] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:19:28.368047 13665 ts_tablet_manager.cc:1918] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:19:28.368211 13667 raft_consensus.cc:2258] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:19:28.368374 13665 log.cc:1196] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/0f149c8a30474092ad81b6b811db83e7
I20260925 12:19:28.368749 13667 ts_tablet_manager.cc:1905] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:19:28.370577 13667 ts_tablet_manager.cc:1918] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:19:28.370915 13665 ts_tablet_manager.cc:1939] T 0f149c8a30474092ad81b6b811db83e7 P d4cd76b08ed940a38ecc13ef540f615b: Deleting consensus metadata
I20260925 12:19:28.371336 13118 catalog_manager.cc:4841] TS d4cd76b08ed940a38ecc13ef540f615b (127.0.98.3:38251): tablet 0f149c8a30474092ad81b6b811db83e7 (table test-table [id=61ba012acffa48978a57c7056101a422]) successfully deleted
I20260925 12:19:28.371517 13667 log.cc:1196] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/0f149c8a30474092ad81b6b811db83e7
I20260925 12:19:28.371851 13667 ts_tablet_manager.cc:1939] T 0f149c8a30474092ad81b6b811db83e7 P c6d96969615f44b58c3780394b1d62ef: Deleting consensus metadata
I20260925 12:19:28.372282 13116 catalog_manager.cc:4841] TS c6d96969615f44b58c3780394b1d62ef (127.0.98.2:41077): tablet 0f149c8a30474092ad81b6b811db83e7 (table test-table [id=61ba012acffa48978a57c7056101a422]) successfully deleted
W20260925 12:19:28.372515 13117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40473 (error 108)
I20260925 12:19:28.373399 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:28.373283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40473 (local address 127.0.0.1:56106)
0925 12:19:28.373326 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:28.373367 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 12:19:28.373394 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 13333
W20260925 12:19:28.373517 13117 catalog_manager.cc:4584] TS 079d3df727204008aaf040ec419da761 (127.0.98.1:40473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f149c8a30474092ad81b6b811db83e7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40473: connect: Connection refused (error 111)
I20260925 12:19:28.379446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 13467
I20260925 12:19:28.384733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 13099
I20260925 12:19:28.391292 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 12:19:28.735512 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 12:19:28.735649 392 mini_ranger.cc:66] Stopping Ranger...
I20260925 12:19:29.079840 392 mini_ranger.cc:68] Stopped Ranger
2026-09-25 12:19:29.079 UTC [12166] LOG: received smart shutdown request
2026-09-25 12:19:29.083 UTC [12166] LOG: background worker "logical replication launcher" (PID 12175) exited with exit code 1
2026-09-25 12:19:29.083 UTC [12170] LOG: shutting down
2026-09-25 12:19:29.229 UTC [12166] LOG: database system is shut down
2026-09-25T12:19:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (44913 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:19:29 dist-test-slave-clmx krb5kdc[13677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:19:29 dist-test-slave-clmx krb5kdc[13677](info): set up 2 sockets
Sep 25 12:19:29 dist-test-slave-clmx krb5kdc[13677](info): commencing operation
krb5kdc: starting...
W20260925 12:19:30.335342 392 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:19:30 dist-test-slave-clmx krb5kdc[13677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:19:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:19:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:19:30.440590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43249
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43249,127.0.98.61:43861,127.0.98.60:46539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:19:30.515308 13690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:30.515516 13690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:30.515552 13690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:30.516670 13690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:19:30.516731 13690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:30.516754 13690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:30.516775 13690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:19:30.516801 13690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:19:30.518181 13690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43249,127.0.98.61:43861,127.0.98.60:46539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:30.518427 13690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:30.518632 13690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:30.523005 13698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:30.523013 13696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:30.523008 13695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:30.523058 13690 server_base.cc:1034] running on GCE node
I20260925 12:19:30.523602 13690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:30.523949 13690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:30.525082 13690 hybrid_clock.cc:648] HybridClock initialized: now 1790338770525070 us; error 30 us; skew 500 ppm
Sep 25 12:19:30 dist-test-slave-clmx krb5kdc[13677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:30.527222 13690 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:19:30.527596 13690 webserver.cc:460] Webserver started at http://127.0.98.62:39357/ using document root <none> and password file <none>
I20260925 12:19:30.527842 13690 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:30.527884 13690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:30.528013 13690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:19:30.528909 13690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "2e8fd269ec894e2baa7cdcf24a3f6346"
format_stamp: "Formatted at 2026-09-25 12:19:30 on dist-test-slave-clmx"
server_key: "51924fb57f43631e4e9f39f77ea0a97f"
server_key_iv: "e0a7e9b9923926381bb83d610739043e"
server_key_version: "encryptionkey@0"
I20260925 12:19:30.529258 13690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e8fd269ec894e2baa7cdcf24a3f6346"
format_stamp: "Formatted at 2026-09-25 12:19:30 on dist-test-slave-clmx"
server_key: "51924fb57f43631e4e9f39f77ea0a97f"
server_key_iv: "e0a7e9b9923926381bb83d610739043e"
server_key_version: "encryptionkey@0"
I20260925 12:19:30.530426 13690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:19:30.531059 13705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:30.531215 13690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:30.531306 13690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "2e8fd269ec894e2baa7cdcf24a3f6346"
format_stamp: "Formatted at 2026-09-25 12:19:30 on dist-test-slave-clmx"
server_key: "51924fb57f43631e4e9f39f77ea0a97f"
server_key_iv: "e0a7e9b9923926381bb83d610739043e"
server_key_version: "encryptionkey@0"
I20260925 12:19:30.531380 13690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:30.548066 13690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:30.549944 13690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:30.550110 13690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:30.553786 13690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43249
I20260925 12:19:30.553828 13757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43249 every 8 connection(s)
I20260925 12:19:30.554131 13690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:19:30.554272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 13690
I20260925 12:19:30.554339 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:19:30.554551 392 external_mini_cluster.cc:1428] Setting key 7bb8659f5569493464b513dd548a8355
I20260925 12:19:30.555068 13758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:19:30.556643 13758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 25 12:19:30 dist-test-slave-clmx krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:30.568742 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.557163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43249 (local address 127.0.0.1:34516)
0925 12:19:30.557325 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.557360 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:30.557914 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:30.558183 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:30.558185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:30.558496 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:30.558700 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:30.558705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.559585 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.559587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:30.560614 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:30.560617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.561077 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.561078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:30.561166 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.561581 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:30.561591 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:30.564916 (+ 3325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:30.567339 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:30.567345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:30.567346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:30.567532 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:30.567792 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:30.567793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:30.567794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:30.567871 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:30.568185 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:30.568189 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:30.568244 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 12:19:30.568346 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 12:19:30.568556 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":102,"thread_start_us":67,"threads_started":1}
I20260925 12:19:30.568742 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.557220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34516 (local address 127.0.98.62:43249)
0925 12:19:30.557335 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:30.557342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:30.557448 (+ 106us) server_negotiation.cc:407] Connection header received
0925 12:19:30.558042 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:30.558044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:30.558058 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:30.558142 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:30.558789 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.559502 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.560693 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.560983 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.561051 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.565004 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:30.565011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:30.565011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:30.565018 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:30.567216 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.567653 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.567659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.567660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.567724 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.567934 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.567936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.567936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.568116 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:30.568204 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:30.568368 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 12:19:30.568519 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":89,"thread_start_us":49,"threads_started":1}
I20260925 12:19:30.569330 13758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:19:30.569636 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.569513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43861 (local address 127.0.0.1:36670)
0925 12:19:30.569564 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.569609 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 12:19:30.569761 13706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43861: connect: Connection refused (error 111)
I20260925 12:19:30.570143 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.570050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43861 (local address 127.0.0.1:36686)
0925 12:19:30.570094 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.570121 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 12:19:30.570254 13758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43861: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43861: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 12:19:30.577548 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43861
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43249,127.0.98.61:43861,127.0.98.60:46539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:19:30.619462 13758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } attempt: 1
I20260925 12:19:30.619892 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.619776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43861 (local address 127.0.0.1:36698)
0925 12:19:30.619833 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.619867 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 12:19:30.620230 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.620146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43861 (local address 127.0.0.1:36708)
0925 12:19:30.620187 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.620212 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 12:19:30.620342 13758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43861: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43861: connect: Connection refused (error 111)
W20260925 12:19:30.652053 13765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:30.652217 13765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:30.652238 13765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:30.653275 13765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:19:30.653306 13765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:30.653321 13765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:30.653331 13765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:19:30.653342 13765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:19:30.654662 13765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43249,127.0.98.61:43861,127.0.98.60:46539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:30.654821 13765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:30.654956 13765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:30.659035 13770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:30.659070 13771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:30.659067 13773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:30.659576 13765 server_base.cc:1034] running on GCE node
I20260925 12:19:30.659792 13765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:30.659979 13765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:30.661100 13765 hybrid_clock.cc:648] HybridClock initialized: now 1790338770661091 us; error 25 us; skew 500 ppm
Sep 25 12:19:30 dist-test-slave-clmx krb5kdc[13677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:30.663281 13765 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 12:19:30.663669 13765 webserver.cc:460] Webserver started at http://127.0.98.61:40733/ using document root <none> and password file <none>
I20260925 12:19:30.663848 13765 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:30.663887 13765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:30.663965 13765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:19:30.664840 13765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-1/data/instance:
uuid: "1db14cc4ea5e4580823f5230df8c444c"
format_stamp: "Formatted at 2026-09-25 12:19:30 on dist-test-slave-clmx"
server_key: "2dd203b7f2c32acbe7d031c02538efda"
server_key_iv: "cd4a091facba04ad7caf663139a9739d"
server_key_version: "encryptionkey@0"
I20260925 12:19:30.665096 13765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-1/wal/instance:
uuid: "1db14cc4ea5e4580823f5230df8c444c"
format_stamp: "Formatted at 2026-09-25 12:19:30 on dist-test-slave-clmx"
server_key: "2dd203b7f2c32acbe7d031c02538efda"
server_key_iv: "cd4a091facba04ad7caf663139a9739d"
server_key_version: "encryptionkey@0"
I20260925 12:19:30.666189 13765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 12:19:30.666764 13780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:30.666949 13765 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:30.667028 13765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-1/wal
uuid: "1db14cc4ea5e4580823f5230df8c444c"
format_stamp: "Formatted at 2026-09-25 12:19:30 on dist-test-slave-clmx"
server_key: "2dd203b7f2c32acbe7d031c02538efda"
server_key_iv: "cd4a091facba04ad7caf663139a9739d"
server_key_version: "encryptionkey@0"
I20260925 12:19:30.667074 13765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:30.676489 13765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:30.678509 13765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:30.678608 13765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:30.682191 13765 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43861
I20260925 12:19:30.682237 13832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43861 every 8 connection(s)
I20260925 12:19:30.682510 13765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-1/data/info.pb
I20260925 12:19:30.683280 13833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:19:30.684589 13833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:19:30.688482 13758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } attempt: 2
Sep 25 12:19:30 dist-test-slave-clmx krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:30.691255 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 13765
I20260925 12:19:30.691342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-1/wal/instance
I20260925 12:19:30.691499 392 external_mini_cluster.cc:1428] Setting key 07f8299dd8e900e1cdfa1bea0f12c5f0
I20260925 12:19:30.693061 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.685209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34526 (local address 127.0.98.62:43249)
0925 12:19:30.685266 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:30.685267 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:30.685451 (+ 184us) server_negotiation.cc:407] Connection header received
0925 12:19:30.685824 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:30.685825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:30.685835 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:30.685879 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:30.686350 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.686911 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.687781 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.687969 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.688023 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.689887 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:30.689888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:30.689888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:30.689895 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:30.691258 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.691729 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.691730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.691730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.692044 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.692325 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.692327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.692327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.692466 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:30.692529 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:30.692891 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 12:19:30.692979 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 12:19:30 dist-test-slave-clmx krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 12:19:30.693449 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.685198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43249 (local address 127.0.0.1:34526)
0925 12:19:30.685341 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.685375 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:30.685754 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:30.685928 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:30.685931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:30.686140 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:30.686291 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:30.686294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.686992 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.686992 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:30.687717 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:30.687719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.688037 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.688037 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:30.688123 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.688464 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:30.688471 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:30.689815 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:30.691336 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:30.691342 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:30.691345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:30.691599 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:30.692122 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:30.692123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:30.692123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:30.692248 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:30.692567 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:30.692573 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:30.692724 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 12:19:30.693160 (+ 436us) client_negotiation.cc:240] Negotiation successful
0925 12:19:30.693309 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260925 12:19:30.694115 13833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:19:30.697261 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.688686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43861 (local address 127.0.0.1:36720)
0925 12:19:30.688734 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.688743 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:30.688780 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:30.689044 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:30.689045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:30.689179 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:30.689218 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:30.689219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.690157 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.690158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:30.690918 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:30.690921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.691550 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.691551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:30.691662 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.692017 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:30.692021 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:30.693451 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:30.695654 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:30.695655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:30.695655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:30.695883 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:30.696108 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:30.696109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:30.696109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:30.696176 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:30.696987 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:30.696987 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:30.697030 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 12:19:30.697080 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 12:19:30.697167 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 12:19:30.697728 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.688756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36720 (local address 127.0.98.61:43861)
0925 12:19:30.688881 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:30.688884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:30.688899 (+ 15us) server_negotiation.cc:407] Connection header received
0925 12:19:30.688914 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:30.688916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:30.688930 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:30.689029 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:30.689298 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.690049 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.691020 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.691466 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.691552 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.693543 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:30.693550 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:30.693550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:30.693558 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:30.695535 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.695974 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.695977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.695977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.696034 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.696737 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.696737 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.696738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.696901 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:30.697474 (+ 573us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:30.697552 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 12:19:30.697632 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":78,"threads_started":1}
Sep 25 12:19:30 dist-test-slave-clmx krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 12:19:30.698266 13758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:19:30.698602 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.698469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46539 (local address 127.0.0.1:49502)
0925 12:19:30.698521 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.698560 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 12:19:30.698889 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.698814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46539 (local address 127.0.0.1:49510)
0925 12:19:30.698845 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.698866 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260925 12:19:30.698961 13758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46539: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46539: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260925 12:19:30.701056 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.694446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43861 (local address 127.0.0.1:36724)
0925 12:19:30.694595 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.694603 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:30.694639 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:30.694789 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:30.694789 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:30.694944 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:30.694965 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:30.694967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.695634 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.695636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:30.696162 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:30.696164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.696695 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.696696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:30.696733 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.697091 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:30.697101 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:30.698323 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:30.700273 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:30.700273 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:30.700274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:30.700406 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:30.700553 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:30.700553 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:30.700553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:30.700589 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:30.700809 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:30.700809 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:30.700870 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 12:19:30.700923 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 12:19:30.700989 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260925 12:19:30.701128 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.694496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36724 (local address 127.0.98.61:43861)
0925 12:19:30.694674 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:30.694676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:30.694683 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:19:30.694699 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:30.694700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:30.694707 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:30.694788 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:30.695035 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.695539 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.696243 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.696625 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.696694 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.698441 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:30.698442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:30.698442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:30.698447 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:30.700206 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.700464 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.700464 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.700464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.700516 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.700665 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.700665 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.700665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.700754 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:30.700797 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:30.700953 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 12:19:30.701055 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
I20260925 12:19:30.701388 13833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:19:30.701735 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.701602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46539 (local address 127.0.0.1:49518)
0925 12:19:30.701649 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.701711 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 12:19:30.701854 13781 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46539: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260925 12:19:30.702229 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.702156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46539 (local address 127.0.0.1:49522)
0925 12:19:30.702187 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.702209 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260925 12:19:30.702361 13833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46539: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46539: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 12:19:30.714323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46539
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43249,127.0.98.61:43861,127.0.98.60:46539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:19:30.742100 13758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } attempt: 1
I20260925 12:19:30.742475 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.742372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46539 (local address 127.0.0.1:49530)
0925 12:19:30.742424 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.742454 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 12:19:30.742807 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.742731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46539 (local address 127.0.0.1:49538)
0925 12:19:30.742770 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.742793 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 12:19:30.742904 13758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46539: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46539: connect: Connection refused (error 111)
I20260925 12:19:30.751567 13833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } attempt: 1
I20260925 12:19:30.751943 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.751838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46539 (local address 127.0.0.1:49550)
0925 12:19:30.751891 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.751921 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 12:19:30.752277 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.752199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46539 (local address 127.0.0.1:49564)
0925 12:19:30.752240 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.752262 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 12:19:30.752384 13833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46539: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46539: connect: Connection refused (error 111)
I20260925 12:19:30.790061 13758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } attempt: 2
I20260925 12:19:30.790421 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.790309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46539 (local address 127.0.0.1:49566)
0925 12:19:30.790366 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.790397 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 12:19:30.790396 13841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:30.790648 13841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:30.790694 13841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 12:19:30.790766 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.790684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46539 (local address 127.0.0.1:49576)
0925 12:19:30.790715 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.790739 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260925 12:19:30.790853 13758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46539: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46539: connect: Connection refused (error 111)
W20260925 12:19:30.791913 13841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:19:30.791952 13841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:30.791967 13841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:30.792022 13841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:19:30.792044 13841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:19:30.793457 13841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43249,127.0.98.61:43861,127.0.98.60:46539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:30.793691 13841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:30.793913 13841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:30.798457 13847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:30.798458 13849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:30.798482 13846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:30.798488 13841 server_base.cc:1034] running on GCE node
I20260925 12:19:30.798836 13841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:30.799104 13841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:30.800287 13841 hybrid_clock.cc:648] HybridClock initialized: now 1790338770800268 us; error 31 us; skew 500 ppm
Sep 25 12:19:30 dist-test-slave-clmx krb5kdc[13677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:30.802558 13841 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 12:19:30.803026 13841 webserver.cc:460] Webserver started at http://127.0.98.60:35573/ using document root <none> and password file <none>
I20260925 12:19:30.803273 13841 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:30.803346 13841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:30.803485 13841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:19:30.804512 13841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-2/data/instance:
uuid: "69a4bea210434a2da5f4ec75678fd46e"
format_stamp: "Formatted at 2026-09-25 12:19:30 on dist-test-slave-clmx"
server_key: "e53de0d7aecd8ebc6e64c1157d39ff82"
server_key_iv: "c3c89c64ee89211bab71082d60d61885"
server_key_version: "encryptionkey@0"
I20260925 12:19:30.804849 13841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-2/wal/instance:
uuid: "69a4bea210434a2da5f4ec75678fd46e"
format_stamp: "Formatted at 2026-09-25 12:19:30 on dist-test-slave-clmx"
server_key: "e53de0d7aecd8ebc6e64c1157d39ff82"
server_key_iv: "c3c89c64ee89211bab71082d60d61885"
server_key_version: "encryptionkey@0"
I20260925 12:19:30.805979 13841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:19:30.806588 13856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:30.806736 13841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:19:30.806823 13841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-2/wal
uuid: "69a4bea210434a2da5f4ec75678fd46e"
format_stamp: "Formatted at 2026-09-25 12:19:30 on dist-test-slave-clmx"
server_key: "e53de0d7aecd8ebc6e64c1157d39ff82"
server_key_iv: "c3c89c64ee89211bab71082d60d61885"
server_key_version: "encryptionkey@0"
I20260925 12:19:30.806895 13841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:30.820539 13833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } attempt: 2
I20260925 12:19:30.820853 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.820753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46539 (local address 127.0.0.1:49584)
0925 12:19:30.820803 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.820832 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 12:19:30.821130 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.821046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46539 (local address 127.0.0.1:49594)
0925 12:19:30.821086 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.821106 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 12:19:30.821198 13833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46539: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46539: connect: Connection refused (error 111)
I20260925 12:19:30.822432 13841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:30.824138 13841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:30.824287 13841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:30.827772 13841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46539
I20260925 12:19:30.827796 13908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46539 every 8 connection(s)
I20260925 12:19:30.828130 13841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-2/data/info.pb
I20260925 12:19:30.828869 13909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:19:30.830178 13909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:19:30 dist-test-slave-clmx krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:30.838248 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.830729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34540 (local address 127.0.98.62:43249)
0925 12:19:30.830768 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:30.830770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:30.830947 (+ 177us) server_negotiation.cc:407] Connection header received
0925 12:19:30.831318 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:30.831319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:30.831327 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:30.831369 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:30.831962 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.832525 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.833790 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.834079 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.834139 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.835935 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:30.835936 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:30.835936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:30.835942 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:30.837236 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.837537 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.837538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.837538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.837580 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.837782 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.837782 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.837782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.837865 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:30.837909 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:30.838151 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 12:19:30.838198 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 12:19:30.838228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 13841
I20260925 12:19:30.838330 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338625091267-392-0/minicluster-data/master-2/wal/instance
I20260925 12:19:30.838466 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.830691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43249 (local address 127.0.0.1:34540)
0925 12:19:30.830841 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.830873 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:30.831251 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:30.831421 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:30.831423 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:30.831714 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:30.831894 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:30.831898 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.832639 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.832641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:30.833719 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:30.833721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.834155 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.834155 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:30.834247 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.834575 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:30.834582 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:30.835859 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:30.837295 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:30.837299 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:30.837301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:30.837473 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:30.837650 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:30.837650 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:30.837651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:30.837734 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:30.837913 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:30.837917 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:30.838011 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 12:19:30.838118 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 12:19:30.838268 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260925 12:19:30.838488 392 external_mini_cluster.cc:1428] Setting key cf17cafd84e7a496444eeb3f5713d5a8
I20260925 12:19:30.839006 13909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:19:30 dist-test-slave-clmx krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 25 12:19:30 dist-test-slave-clmx krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 12:19:30.845083 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.839242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43861 (local address 127.0.0.1:36726)
0925 12:19:30.839285 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.839294 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:30.839325 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:30.839442 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:30.839442 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:30.839563 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:30.839602 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:30.839603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.840223 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.840224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:30.840655 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:30.840656 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.841003 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.841004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:30.841066 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.841375 (+ 309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:30.841379 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:30.842560 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:30.844062 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:30.844063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:30.844063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:30.844286 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:30.844494 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:30.844495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:30.844495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:30.844571 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:30.844795 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:30.844796 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:30.844838 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 12:19:30.844902 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 12:19:30.844988 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 12:19:30.845099 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.839311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36726 (local address 127.0.98.61:43861)
0925 12:19:30.839356 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:30.839358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:30.839366 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:19:30.839386 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:30.839387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:30.839396 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:30.839441 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:30.839692 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.840134 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.840736 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.840931 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.840999 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.842640 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:30.842641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:30.842641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:30.842648 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:30.843994 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.844370 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.844371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.844371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.844431 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.844637 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.844638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.844638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.844731 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:30.844781 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:30.844944 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 12:19:30.845025 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:19:30.845409 13909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:19:30.847059 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.838764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49596 (local address 127.0.98.60:46539)
0925 12:19:30.838887 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:30.838891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:30.839040 (+ 149us) server_negotiation.cc:407] Connection header received
0925 12:19:30.839059 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:30.839060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:30.839072 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:30.839150 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:30.839842 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.840437 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.841122 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.841532 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.841627 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.844017 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:30.844024 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:30.844025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:30.844032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:30.845865 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.846240 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.846242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.846242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.846289 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.846481 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.846481 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.846481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.846591 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:30.846722 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:30.846784 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 12:19:30.846834 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
Sep 25 12:19:30 dist-test-slave-clmx krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 12:19:30 dist-test-slave-clmx krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:30.852267 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.845640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46539 (local address 127.0.0.1:49604)
0925 12:19:30.845680 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.845689 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:30.845722 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:30.845927 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:30.845928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:30.846045 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:30.846068 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:30.846069 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.846883 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.846884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:30.847483 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:30.847485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.848093 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.848093 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:30.848145 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.848457 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:30.848462 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:30.849537 (+ 1075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:30.851215 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:30.851216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:30.851217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:30.851403 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:30.851590 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:30.851592 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:30.851592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:30.851673 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:30.851916 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:30.851917 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:30.851960 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 12:19:30.852032 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 12:19:30.852129 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 12:19:30.852269 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.845673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49604 (local address 127.0.98.60:46539)
0925 12:19:30.845806 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:30.845808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:30.845815 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:19:30.845832 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:30.845833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:30.845847 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:30.845902 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:30.846136 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.846813 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.847593 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.847983 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.848059 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.849633 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:30.849634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:30.849634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:30.849640 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:30.851125 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.851471 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.851472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.851472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.851520 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.851752 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.851753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.851753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.851837 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:30.851910 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:30.852057 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 12:19:30.852174 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":112,"thread_start_us":91,"threads_started":1}
I20260925 12:19:30.853724 13909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e: Bootstrap starting.
I20260925 12:19:30.854274 13909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e: Neither blocks nor log segments found. Creating new log.
I20260925 12:19:30.854547 13909 log.cc:824] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:30.854858 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.847833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34546 (local address 127.0.98.62:43249)
0925 12:19:30.847861 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:30.847879 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:30.847886 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:19:30.847942 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:30.847943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:30.847951 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:30.848012 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:30.848646 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.849034 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.849754 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.849942 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.850021 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.852346 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:30.852347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:30.852348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:30.852353 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:30.853735 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.854095 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.854096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.854097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.854146 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.854361 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.854362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.854362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.854462 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:30.854652 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:30.854689 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 12:19:30.854754 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 12:19:30.855403 13909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e: No bootstrap required, opened a new log
I20260925 12:19:30.857157 13909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } peers { permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } } peers { permanent_uuid: "69a4bea210434a2da5f4ec75678fd46e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } }
I20260925 12:19:30.857278 13909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:19:30.857303 13909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a4bea210434a2da5f4ec75678fd46e, State: Initialized, Role: FOLLOWER
I20260925 12:19:30.857400 13909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } peers { permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } } peers { permanent_uuid: "69a4bea210434a2da5f4ec75678fd46e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } }
I20260925 12:19:30.857671 13909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:19:30.857640 13916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } peers { permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } } peers { permanent_uuid: "69a4bea210434a2da5f4ec75678fd46e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } } }
I20260925 12:19:30.857748 13916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [sys.catalog]: This master's current role is: FOLLOWER
Sep 25 12:19:30 dist-test-slave-clmx krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260925 12:19:30.859201 13927 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:19:30.859691 13927 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:19:30.860319 13841 master_runner.cc:186] Error getting master registration for 127.0.98.62:43249: OK, instance_id { permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" instance_seqno: 1790338770552778 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 12:19:30.862264 13841 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 12:19:30.862272 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.855505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36736 (local address 127.0.98.61:43861)
0925 12:19:30.855542 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:30.855544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:30.855600 (+ 56us) server_negotiation.cc:407] Connection header received
0925 12:19:30.855692 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:30.855693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:30.855701 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:30.855750 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:30.856277 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.856645 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.857435 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.857698 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.857766 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.859864 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:30.859865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:30.859865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:30.859871 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:30.861154 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.861499 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.861500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.861500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.861555 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.861777 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.861778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.861778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.861882 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:30.862024 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:30.862101 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 12:19:30.862168 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:19:30.863598 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:19:30.873023 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.865039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34556 (local address 127.0.98.62:43249)
0925 12:19:30.865075 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:30.865077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:30.865261 (+ 184us) server_negotiation.cc:407] Connection header received
0925 12:19:30.865282 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:30.865282 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:30.865290 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:30.865348 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:30.866009 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.866645 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.867702 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.867970 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.868030 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.869511 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:30.869512 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:30.869513 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:30.869519 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:30.871162 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.871518 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.871519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.871520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.871574 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.871838 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.871838 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.871839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.871953 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:30.872496 (+ 543us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:30.872851 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 12:19:30.872940 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:19:30.873068 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.865234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.0.98.61:43861)
0925 12:19:30.865272 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:30.865274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:30.865368 (+ 94us) server_negotiation.cc:407] Connection header received
0925 12:19:30.865462 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:30.865463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:30.865471 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:30.865500 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:30.866190 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.866837 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.867849 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.868126 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.868162 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.869772 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:30.869773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:30.869773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:30.869779 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:30.871909 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.872291 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.872292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.872293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.872342 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.872546 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.872547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.872547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.872655 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:30.872713 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:30.872921 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 12:19:30.872988 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:19:30.873521 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.865467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49608 (local address 127.0.98.60:46539)
0925 12:19:30.865498 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:30.865500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:30.865589 (+ 89us) server_negotiation.cc:407] Connection header received
0925 12:19:30.865677 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:30.865677 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:30.865686 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:30.865713 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:30.866350 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.866956 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.867906 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.868242 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.868720 (+ 478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.870455 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:30.870456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:30.870456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:30.870462 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:30.872265 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.872674 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.872675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.872676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.872728 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.872943 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.872944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.872944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.873090 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:30.873169 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:30.873364 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 12:19:30.873433 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 12:19:30.898083 13758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } attempt: 3
Sep 25 12:19:30 dist-test-slave-clmx krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 12:19:30.904445 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.898406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49624 (local address 127.0.98.60:46539)
0925 12:19:30.898465 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:30.898467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:30.898475 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:19:30.898524 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:30.898525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:30.898536 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:30.898598 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:30.898843 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.899280 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.899960 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.900131 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.900197 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.902034 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:30.902035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:30.902036 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:30.902042 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:30.903354 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.903785 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.903786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.903786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.903832 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.904015 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.904016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.904016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.904105 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:30.904149 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:30.904331 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 12:19:30.904392 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 12:19:30.904517 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.898358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46539 (local address 127.0.0.1:49624)
0925 12:19:30.898410 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.898418 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:30.898471 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:30.898601 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:30.898602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:30.898738 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:30.898771 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:30.898772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.899365 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.899367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:30.899889 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:30.899891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.900205 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.900206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:30.900262 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.900662 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:30.900667 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:30.901942 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:30.903475 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:30.903475 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:30.903476 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:30.903714 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:30.903894 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:30.903895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:30.903895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:30.903958 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:30.904162 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:30.904163 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:30.904217 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 12:19:30.904319 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 12:19:30.904416 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 12:19:30.905916 13758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346: Bootstrap starting.
I20260925 12:19:30.906487 13758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346: Neither blocks nor log segments found. Creating new log.
I20260925 12:19:30.906767 13758 log.cc:824] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:30.907454 13758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346: No bootstrap required, opened a new log
I20260925 12:19:30.908864 13758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } peers { permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } } peers { permanent_uuid: "69a4bea210434a2da5f4ec75678fd46e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } }
I20260925 12:19:30.909004 13758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:19:30.909052 13758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e8fd269ec894e2baa7cdcf24a3f6346, State: Initialized, Role: FOLLOWER
I20260925 12:19:30.909178 13758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } peers { permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } } peers { permanent_uuid: "69a4bea210434a2da5f4ec75678fd46e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } }
I20260925 12:19:30.909428 13937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } peers { permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } } peers { permanent_uuid: "69a4bea210434a2da5f4ec75678fd46e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } } }
I20260925 12:19:30.909485 13758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:19:30.909502 13937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 12:19:30.911216 13948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:19:30.911331 13948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:19:30.912331 13833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } attempt: 3
I20260925 12:19:30.912002 13690 master_runner.cc:186] Error getting master registration for 127.0.98.61:43861: OK, instance_id { permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" instance_seqno: 1790338770681247 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 12:19:30.914662 13690 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 25 12:19:30 dist-test-slave-clmx krb5kdc[13677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 12:19:30.919781 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.912721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49640 (local address 127.0.98.60:46539)
0925 12:19:30.912759 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:30.912761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:30.912768 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:19:30.912786 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:30.912787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:30.912796 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:30.912842 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:30.913158 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.913797 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.914758 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.915028 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.915096 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.917023 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:30.917024 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:30.917024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:30.917030 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:30.918561 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.918974 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.918975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.918975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.919044 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:30.919250 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:30.919251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:30.919251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:30.919373 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:30.919441 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:30.919605 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 12:19:30.919698 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:19:30.919793 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:30.912542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46539 (local address 127.0.0.1:49640)
0925 12:19:30.912611 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:30.912621 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:30.912656 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:30.912860 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:30.912861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:30.913012 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:30.913063 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:30.913065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.913889 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.913890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:30.914665 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:30.914668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:30.915143 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:30.915144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:30.915196 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:30.915593 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:30.915597 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:30.916943 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:30.918663 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:30.918664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:30.918665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:30.918867 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:30.919116 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:30.919116 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:30.919117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:30.919181 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:30.919441 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:30.919442 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:30.919485 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 12:19:30.919560 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 12:19:30.919701 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 12:19:30.921573 13833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c: Bootstrap starting.
I20260925 12:19:30.922246 13833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c: Neither blocks nor log segments found. Creating new log.
I20260925 12:19:30.922515 13833 log.cc:824] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:30.923697 13833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c: No bootstrap required, opened a new log
I20260925 12:19:30.925473 13833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } peers { permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } } peers { permanent_uuid: "69a4bea210434a2da5f4ec75678fd46e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } }
I20260925 12:19:30.925575 13833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:19:30.925603 13833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1db14cc4ea5e4580823f5230df8c444c, State: Initialized, Role: FOLLOWER
I20260925 12:19:30.925699 13833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } peers { permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } } peers { permanent_uuid: "69a4bea210434a2da5f4ec75678fd46e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } }
I20260925 12:19:30.925933 13833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:19:30.925987 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } peers { permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } } peers { permanent_uuid: "69a4bea210434a2da5f4ec75678fd46e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } } }
I20260925 12:19:30.926128 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c [sys.catalog]: This master's current role is: FOLLOWER
W20260925 12:19:30.927778 13962 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:19:30.927881 13962 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260925 12:19:30.929069 13786 tablet.cc:2369] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:19:30.929481 13765 master_runner.cc:263] No leader master found from master 1db14cc4ea5e4580823f5230df8c444c
I20260925 12:19:30.929519 13765 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 12:19:30.972676 13916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:19:30.972792 13916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } peers { permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } } peers { permanent_uuid: "69a4bea210434a2da5f4ec75678fd46e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } }
I20260925 12:19:30.973011 13916 leader_election.cc:288] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e8fd269ec894e2baa7cdcf24a3f6346 (127.0.98.62:43249), 1db14cc4ea5e4580823f5230df8c444c (127.0.98.61:43861)
I20260925 12:19:30.973328 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "69a4bea210434a2da5f4ec75678fd46e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1db14cc4ea5e4580823f5230df8c444c" is_pre_election: true
I20260925 12:19:30.973464 13807 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69a4bea210434a2da5f4ec75678fd46e in term 0.
I20260925 12:19:30.973640 13857 leader_election.cc:302] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1db14cc4ea5e4580823f5230df8c444c, 69a4bea210434a2da5f4ec75678fd46e; no voters:
I20260925 12:19:30.973965 13732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "69a4bea210434a2da5f4ec75678fd46e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" is_pre_election: true
I20260925 12:19:30.974097 13732 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69a4bea210434a2da5f4ec75678fd46e in term 0.
I20260925 12:19:30.974367 13916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:19:30.974421 13916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:19:30.974445 13916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:19:30.975278 13916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } peers { permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } } peers { permanent_uuid: "69a4bea210434a2da5f4ec75678fd46e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } }
I20260925 12:19:30.975378 13916 leader_election.cc:288] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [CANDIDATE]: Term 1 election: Requested vote from peers 2e8fd269ec894e2baa7cdcf24a3f6346 (127.0.98.62:43249), 1db14cc4ea5e4580823f5230df8c444c (127.0.98.61:43861)
I20260925 12:19:30.975673 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "69a4bea210434a2da5f4ec75678fd46e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1db14cc4ea5e4580823f5230df8c444c"
I20260925 12:19:30.975740 13807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:19:30.976606 13807 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69a4bea210434a2da5f4ec75678fd46e in term 1.
I20260925 12:19:30.976799 13857 leader_election.cc:302] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1db14cc4ea5e4580823f5230df8c444c, 69a4bea210434a2da5f4ec75678fd46e; no voters:
I20260925 12:19:30.976948 13732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "69a4bea210434a2da5f4ec75678fd46e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346"
I20260925 12:19:30.977010 13732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:19:30.977741 13732 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69a4bea210434a2da5f4ec75678fd46e in term 1.
I20260925 12:19:30.978325 13916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:19:30.978490 13916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [term 1 LEADER]: Becoming Leader. State: Replica: 69a4bea210434a2da5f4ec75678fd46e, State: Running, Role: LEADER
I20260925 12:19:30.978569 13916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } peers { permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } } peers { permanent_uuid: "69a4bea210434a2da5f4ec75678fd46e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } }
I20260925 12:19:30.978891 13916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69a4bea210434a2da5f4ec75678fd46e. Latest consensus state: current_term: 1 leader_uuid: "69a4bea210434a2da5f4ec75678fd46e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } peers { permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } } peers { permanent_uuid: "69a4bea210434a2da5f4ec75678fd46e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } } }
I20260925 12:19:30.978955 13916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [sys.catalog]: This master's current role is: LEADER
I20260925 12:19:30.979933 13966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:19:30.980151 13966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:19:30.981601 13807 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c [term 1 FOLLOWER]: Refusing update from remote peer 69a4bea210434a2da5f4ec75678fd46e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:19:30.981968 13916 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:19:30.983239 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69a4bea210434a2da5f4ec75678fd46e. Latest consensus state: current_term: 1 leader_uuid: "69a4bea210434a2da5f4ec75678fd46e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } peers { permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } } peers { permanent_uuid: "69a4bea210434a2da5f4ec75678fd46e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } } }
I20260925 12:19:30.983331 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:19:30.983747 13732 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346 [term 1 FOLLOWER]: Refusing update from remote peer 69a4bea210434a2da5f4ec75678fd46e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:19:30.984019 13916 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:19:30.985178 13937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69a4bea210434a2da5f4ec75678fd46e. Latest consensus state: current_term: 1 leader_uuid: "69a4bea210434a2da5f4ec75678fd46e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } peers { permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } } peers { permanent_uuid: "69a4bea210434a2da5f4ec75678fd46e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } } }
I20260925 12:19:30.985257 13937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:19:30.986431 13964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "69a4bea210434a2da5f4ec75678fd46e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } peers { permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } } peers { permanent_uuid: "69a4bea210434a2da5f4ec75678fd46e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } } }
I20260925 12:19:30.986526 13964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [sys.catalog]: This master's current role is: LEADER
I20260925 12:19:30.986907 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "69a4bea210434a2da5f4ec75678fd46e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } peers { permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } } peers { permanent_uuid: "69a4bea210434a2da5f4ec75678fd46e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } } }
I20260925 12:19:30.986996 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:19:30.987421 13967 mvcc.cc:205] Tried to move back new op lower bound from 7333227605938503680 to 7333227605928669184. Current Snapshot: MvccSnapshot[applied={T|T < 7333227605938503680}]
I20260925 12:19:30.987911 13968 mvcc.cc:205] Tried to move back new op lower bound from 7333227605938503680 to 7333227605928669184. Current Snapshot: MvccSnapshot[applied={T|T < 7333227605938503680}]
I20260925 12:19:30.988417 13966 catalog_manager.cc:1347] Generated new cluster ID: 32e0c91b1be646c394ffd70a99e72241
I20260925 12:19:30.988461 13966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:19:31.000216 13916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "69a4bea210434a2da5f4ec75678fd46e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } peers { permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } } peers { permanent_uuid: "69a4bea210434a2da5f4ec75678fd46e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } } }
I20260925 12:19:31.000286 13916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e [sys.catalog]: This master's current role is: LEADER
I20260925 12:19:31.000635 13937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "69a4bea210434a2da5f4ec75678fd46e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e8fd269ec894e2baa7cdcf24a3f6346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } peers { permanent_uuid: "1db14cc4ea5e4580823f5230df8c444c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43861 } } peers { permanent_uuid: "69a4bea210434a2da5f4ec75678fd46e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46539 } } }
I20260925 12:19:31.000710 13937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:19:31.006955 13966 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:19:31.007670 13966 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:19:31.022109 13966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69a4bea210434a2da5f4ec75678fd46e: Generated new TSK 0
I20260925 12:19:31.022265 13966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:19:31.911775 13948 catalog_manager.cc:1259] Loaded cluster ID: 32e0c91b1be646c394ffd70a99e72241
I20260925 12:19:31.911866 13948 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346: loading cluster ID for follower catalog manager: success
I20260925 12:19:31.912694 13948 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346: acquiring CA information for follower catalog manager: success
I20260925 12:19:31.913002 13948 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2e8fd269ec894e2baa7cdcf24a3f6346: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 12:19:31.928315 13962 catalog_manager.cc:1259] Loaded cluster ID: 32e0c91b1be646c394ffd70a99e72241
I20260925 12:19:31.928402 13962 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c: loading cluster ID for follower catalog manager: success
I20260925 12:19:31.929198 13962 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c: acquiring CA information for follower catalog manager: success
I20260925 12:19:31.929503 13962 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1db14cc4ea5e4580823f5230df8c444c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 12:19:32.074132 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 13690
I20260925 12:19:32.078991 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 13765
I20260925 12:19:32.085563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 13841
2026-09-25T12:19:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2758 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:19:32 dist-test-slave-clmx krb5kdc[13983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:19:32 dist-test-slave-clmx krb5kdc[13983](info): set up 2 sockets
Sep 25 12:19:32 dist-test-slave-clmx krb5kdc[13983](info): commencing operation
krb5kdc: starting...
W20260925 12:19:33.087423 392 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:19:33 dist-test-slave-clmx krb5kdc[13983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:19:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:19:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:19:33.193699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43323
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:19:33.268289 13996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:33.268497 13996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:33.268534 13996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:33.269665 13996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:19:33.269730 13996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:33.269757 13996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:33.269789 13996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:19:33.269812 13996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:19:33.271206 13996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:33.271462 13996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:33.271706 13996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:33.276252 14002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:33.276257 14001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:33.276257 14004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:33.276481 13996 server_base.cc:1034] running on GCE node
I20260925 12:19:33.276746 13996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:33.277045 13996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:33.278180 13996 hybrid_clock.cc:648] HybridClock initialized: now 1790338773278170 us; error 27 us; skew 500 ppm
Sep 25 12:19:33 dist-test-slave-clmx krb5kdc[13983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:33.280645 13996 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:19:33.281090 13996 webserver.cc:460] Webserver started at http://127.0.98.62:42167/ using document root <none> and password file <none>
I20260925 12:19:33.281298 13996 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:33.281363 13996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:33.281508 13996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:19:33.282480 13996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "bff68aeb9b554af79642e1b153428068"
format_stamp: "Formatted at 2026-09-25 12:19:33 on dist-test-slave-clmx"
server_key: "9a30d90a8ab752d2c0c55fced0253810"
server_key_iv: "8a88d0c2ae016b42cb4b5ebd9df6d2a1"
server_key_version: "encryptionkey@0"
I20260925 12:19:33.282829 13996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "bff68aeb9b554af79642e1b153428068"
format_stamp: "Formatted at 2026-09-25 12:19:33 on dist-test-slave-clmx"
server_key: "9a30d90a8ab752d2c0c55fced0253810"
server_key_iv: "8a88d0c2ae016b42cb4b5ebd9df6d2a1"
server_key_version: "encryptionkey@0"
I20260925 12:19:33.284155 13996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:19:33.284838 14011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:33.284983 13996 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:19:33.285077 13996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "bff68aeb9b554af79642e1b153428068"
format_stamp: "Formatted at 2026-09-25 12:19:33 on dist-test-slave-clmx"
server_key: "9a30d90a8ab752d2c0c55fced0253810"
server_key_iv: "8a88d0c2ae016b42cb4b5ebd9df6d2a1"
server_key_version: "encryptionkey@0"
I20260925 12:19:33.285157 13996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:33.328727 13996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:33.331012 13996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:33.331180 13996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:33.334995 13996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43323
I20260925 12:19:33.335026 14063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43323 every 8 connection(s)
I20260925 12:19:33.335346 13996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:19:33.336170 14064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:19:33.338124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 13996
I20260925 12:19:33.338189 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:19:33.338320 14064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068: Bootstrap starting.
I20260925 12:19:33.338349 392 external_mini_cluster.cc:1428] Setting key b01af320a09d78f8eaef75e4fa0f123a
I20260925 12:19:33.339005 14064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068: Neither blocks nor log segments found. Creating new log.
I20260925 12:19:33.339306 14064 log.cc:824] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:33.339984 14064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068: No bootstrap required, opened a new log
I20260925 12:19:33.342166 14064 raft_consensus.cc:348] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff68aeb9b554af79642e1b153428068" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43323 } }
I20260925 12:19:33.342290 14064 raft_consensus.cc:374] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:19:33.342311 14064 raft_consensus.cc:729] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bff68aeb9b554af79642e1b153428068, State: Initialized, Role: FOLLOWER
I20260925 12:19:33.342439 14064 consensus_queue.cc:260] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff68aeb9b554af79642e1b153428068" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43323 } }
I20260925 12:19:33.342519 14064 raft_consensus.cc:388] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:19:33.342550 14064 raft_consensus.cc:482] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:19:33.342594 14064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:19:33.343223 14064 raft_consensus.cc:504] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff68aeb9b554af79642e1b153428068" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43323 } }
I20260925 12:19:33.343324 14064 leader_election.cc:302] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bff68aeb9b554af79642e1b153428068; no voters:
I20260925 12:19:33.343494 14064 leader_election.cc:288] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:19:33.343545 14069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:19:33.343761 14064 sys_catalog.cc:564] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:19:33.344691 14069 raft_consensus.cc:686] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068 [term 1 LEADER]: Becoming Leader. State: Replica: bff68aeb9b554af79642e1b153428068, State: Running, Role: LEADER
I20260925 12:19:33.344786 14069 consensus_queue.cc:237] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff68aeb9b554af79642e1b153428068" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43323 } }
I20260925 12:19:33.345129 14069 sys_catalog.cc:455] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bff68aeb9b554af79642e1b153428068. Latest consensus state: current_term: 1 leader_uuid: "bff68aeb9b554af79642e1b153428068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff68aeb9b554af79642e1b153428068" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43323 } } }
Sep 25 12:19:33 dist-test-slave-clmx krb5kdc[13983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:33.345193 14069 sys_catalog.cc:458] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068 [sys.catalog]: This master's current role is: LEADER
I20260925 12:19:33.345453 14070 sys_catalog.cc:455] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bff68aeb9b554af79642e1b153428068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff68aeb9b554af79642e1b153428068" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43323 } } }
I20260925 12:19:33.345533 14070 sys_catalog.cc:458] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068 [sys.catalog]: This master's current role is: LEADER
I20260925 12:19:33.345606 14075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:19:33.346164 14075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:19:33.347852 14075 catalog_manager.cc:1347] Generated new cluster ID: a7462d6b4eca44b9b0ab6cde7412cf2e
I20260925 12:19:33.347899 14075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:19:33.349152 14067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:33.338791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56656 (local address 127.0.98.62:43323)
0925 12:19:33.338972 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:33.338982 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:33.339013 (+ 31us) server_negotiation.cc:407] Connection header received
0925 12:19:33.339560 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:33.339566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:33.339609 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:33.339796 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:33.340364 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:33.341686 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:33.342509 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:33.343123 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:33.343258 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:33.345712 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:33.345716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:33.345716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:33.345727 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:33.347683 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:33.348043 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:33.348046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:33.348046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:33.348142 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:33.348352 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:33.348352 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:33.348353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:33.348613 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:33.348732 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:33.348867 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 12:19:33.348984 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":72,"threads_started":1}
I20260925 12:19:33.349959 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:19:33.364549 14075 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:19:33.365211 14075 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:19:33.375211 14075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bff68aeb9b554af79642e1b153428068: Generated new TSK 0
I20260925 12:19:33.375358 14075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:19:33.403349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 13996
2026-09-25T12:19:33Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1313 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:19:33 dist-test-slave-clmx krb5kdc[14091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:19:33 dist-test-slave-clmx krb5kdc[14091](info): set up 2 sockets
Sep 25 12:19:33 dist-test-slave-clmx krb5kdc[14091](info): commencing operation
krb5kdc: starting...
W20260925 12:19:34.395654 392 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:19:34 dist-test-slave-clmx krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:19:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:19:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:19:34.500237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32939
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:19:34.574246 14104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:34.574410 14104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:34.574429 14104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:34.575438 14104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:19:34.575475 14104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:34.575488 14104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:34.575500 14104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:19:34.575558 14104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:19:34.576841 14104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:34.576993 14104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:34.577200 14104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:34.581521 14109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:34.581535 14110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:34.581643 14112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:34.581642 14104 server_base.cc:1034] running on GCE node
I20260925 12:19:34.581895 14104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:34.582147 14104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:34.583299 14104 hybrid_clock.cc:648] HybridClock initialized: now 1790338774583284 us; error 28 us; skew 500 ppm
Sep 25 12:19:34 dist-test-slave-clmx krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:34.585587 14104 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:19:34.586001 14104 webserver.cc:460] Webserver started at http://127.0.98.62:34387/ using document root <none> and password file <none>
I20260925 12:19:34.586162 14104 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:34.586198 14104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:34.586278 14104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:19:34.587143 14104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "43c4b6e343b64336b021ab9c29c13f3a"
format_stamp: "Formatted at 2026-09-25 12:19:34 on dist-test-slave-clmx"
server_key: "c1178a865042d05fb76af66a379c8484"
server_key_iv: "eb10a6dd3d57b448e4e3501071f1335c"
server_key_version: "encryptionkey@0"
I20260925 12:19:34.587410 14104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "43c4b6e343b64336b021ab9c29c13f3a"
format_stamp: "Formatted at 2026-09-25 12:19:34 on dist-test-slave-clmx"
server_key: "c1178a865042d05fb76af66a379c8484"
server_key_iv: "eb10a6dd3d57b448e4e3501071f1335c"
server_key_version: "encryptionkey@0"
I20260925 12:19:34.588610 14104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 12:19:34.589179 14119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:34.589316 14104 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:34.589392 14104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "43c4b6e343b64336b021ab9c29c13f3a"
format_stamp: "Formatted at 2026-09-25 12:19:34 on dist-test-slave-clmx"
server_key: "c1178a865042d05fb76af66a379c8484"
server_key_iv: "eb10a6dd3d57b448e4e3501071f1335c"
server_key_version: "encryptionkey@0"
I20260925 12:19:34.589439 14104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:34.596513 14104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:34.597983 14104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:34.598078 14104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:34.601585 14104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32939
I20260925 12:19:34.601605 14171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32939 every 8 connection(s)
I20260925 12:19:34.601930 14104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:19:34.602835 14172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:19:34.603700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 14104
I20260925 12:19:34.603796 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:19:34.604002 392 external_mini_cluster.cc:1428] Setting key eb3da0ac7a68fa759d40dc401db6aeae
I20260925 12:19:34.605850 14172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a: Bootstrap starting.
I20260925 12:19:34.606504 14172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a: Neither blocks nor log segments found. Creating new log.
I20260925 12:19:34.606832 14172 log.cc:824] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:34.607474 14172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a: No bootstrap required, opened a new log
I20260925 12:19:34.609579 14172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4b6e343b64336b021ab9c29c13f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } }
I20260925 12:19:34.609694 14172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:19:34.609711 14172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c4b6e343b64336b021ab9c29c13f3a, State: Initialized, Role: FOLLOWER
I20260925 12:19:34.609846 14172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4b6e343b64336b021ab9c29c13f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } }
I20260925 12:19:34.609925 14172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:19:34.609961 14172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:19:34.610002 14172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 0 FOLLOWER]: Advancing to term 1
Sep 25 12:19:34 dist-test-slave-clmx krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:34.610633 14172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4b6e343b64336b021ab9c29c13f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } }
I20260925 12:19:34.610744 14172 leader_election.cc:302] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43c4b6e343b64336b021ab9c29c13f3a; no voters:
I20260925 12:19:34.610903 14172 leader_election.cc:288] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:19:34.610966 14177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:19:34.611136 14172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:19:34.611162 14177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 1 LEADER]: Becoming Leader. State: Replica: 43c4b6e343b64336b021ab9c29c13f3a, State: Running, Role: LEADER
I20260925 12:19:34.611402 14177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4b6e343b64336b021ab9c29c13f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } }
I20260925 12:19:34.611781 14179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43c4b6e343b64336b021ab9c29c13f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4b6e343b64336b021ab9c29c13f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } } }
I20260925 12:19:34.611862 14179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [sys.catalog]: This master's current role is: LEADER
I20260925 12:19:34.611796 14178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43c4b6e343b64336b021ab9c29c13f3a. Latest consensus state: current_term: 1 leader_uuid: "43c4b6e343b64336b021ab9c29c13f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4b6e343b64336b021ab9c29c13f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } } }
I20260925 12:19:34.612179 14178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [sys.catalog]: This master's current role is: LEADER
I20260925 12:19:34.612975 14186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:19:34.613487 14186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:19:34.614519 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:34.604489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60624 (local address 127.0.98.62:32939)
0925 12:19:34.604646 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:34.604654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:34.604682 (+ 28us) server_negotiation.cc:407] Connection header received
0925 12:19:34.605157 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:34.605166 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:34.605227 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:34.605373 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:34.605936 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:34.607257 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:34.608008 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:34.608567 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:34.608671 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:34.611053 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:34.611057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:34.611057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:34.611067 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:34.612905 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:34.613321 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:34.613323 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:34.613323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:34.613414 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:34.613597 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:34.613598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:34.613598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:34.613890 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:34.614035 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:34.614133 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 12:19:34.614350 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260925 12:19:34.614890 14186 catalog_manager.cc:1347] Generated new cluster ID: 6ea7f03a141b4dd9a9d6a3518892a723
I20260925 12:19:34.614961 14186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:19:34.635080 14186 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:19:34.635754 14186 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:19:34.648067 14186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a: Generated new TSK 0
I20260925 12:19:34.648219 14186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:19:34.676664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32939
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:19:34.750425 14198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:34.750613 14198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:34.750648 14198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:34.751780 14198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:34.751837 14198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:34.751878 14198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:19:34.753257 14198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:34.753507 14198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:34.753711 14198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:34.758210 14206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:34.758311 14198 server_base.cc:1034] running on GCE node
W20260925 12:19:34.758225 14204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:34.758219 14203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:34.758638 14198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:34.758831 14198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:34.759959 14198 hybrid_clock.cc:648] HybridClock initialized: now 1790338774759946 us; error 31 us; skew 500 ppm
Sep 25 12:19:34 dist-test-slave-clmx krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:34.762300 14198 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:19:34.762723 14198 webserver.cc:460] Webserver started at http://127.0.98.1:32897/ using document root <none> and password file <none>
I20260925 12:19:34.762925 14198 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:34.762966 14198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:34.763126 14198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:19:34.764137 14198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "18d3e88caaa34418876c3759d893f8fe"
format_stamp: "Formatted at 2026-09-25 12:19:34 on dist-test-slave-clmx"
server_key: "a1272ee60da76e59fdec3a0e9838de01"
server_key_iv: "ccf7711c75097f1f0057c4e4d3a249f1"
server_key_version: "encryptionkey@0"
I20260925 12:19:34.764473 14198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "18d3e88caaa34418876c3759d893f8fe"
format_stamp: "Formatted at 2026-09-25 12:19:34 on dist-test-slave-clmx"
server_key: "a1272ee60da76e59fdec3a0e9838de01"
server_key_iv: "ccf7711c75097f1f0057c4e4d3a249f1"
server_key_version: "encryptionkey@0"
I20260925 12:19:34.765645 14198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:19:34.766254 14213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:34.766445 14198 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:19:34.766510 14198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "18d3e88caaa34418876c3759d893f8fe"
format_stamp: "Formatted at 2026-09-25 12:19:34 on dist-test-slave-clmx"
server_key: "a1272ee60da76e59fdec3a0e9838de01"
server_key_iv: "ccf7711c75097f1f0057c4e4d3a249f1"
server_key_version: "encryptionkey@0"
I20260925 12:19:34.766592 14198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:34.776337 14198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:34.777756 14198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:34.777901 14198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:34.778084 14198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:19:34.778345 14198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:19:34.778376 14198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:34.778393 14198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:19:34.778424 14198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:34.783844 14198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44365
I20260925 12:19:34.783892 14326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44365 every 8 connection(s)
I20260925 12:19:34.784150 14198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 12:19:34 dist-test-slave-clmx krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:34.790740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 14198
I20260925 12:19:34.790846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:19:34.791049 392 external_mini_cluster.cc:1428] Setting key 8b0d04cc278d4473d7c61024b212f42b
I20260925 12:19:34.792371 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:34.784787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52011 (local address 127.0.98.62:32939)
0925 12:19:34.784849 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:34.784851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:34.784992 (+ 141us) server_negotiation.cc:407] Connection header received
0925 12:19:34.785407 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:34.785408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:34.785416 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:34.785467 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:34.786279 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:34.786752 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:34.787556 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:34.787836 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:34.787883 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:34.789572 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:34.789574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:34.789574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:34.789580 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:34.790752 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:34.791219 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:34.791220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:34.791221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:34.791295 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:34.791713 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:34.791713 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:34.791714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:34.791825 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:34.791929 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:34.792195 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 12:19:34.792279 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 12:19:34.792557 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:34.784755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:52011)
0925 12:19:34.784894 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:34.784929 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:34.785331 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:34.785511 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:34.785513 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:34.786019 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:34.786183 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:34.786186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:34.786817 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:34.786818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:34.787487 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:34.787488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:34.787899 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:34.787900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:34.787986 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:34.788333 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:34.788340 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:34.789487 (+ 1147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:34.790839 (+ 1352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:34.790845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:34.790847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:34.791100 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:34.791495 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:34.791496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:34.791496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:34.791654 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:34.791925 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:34.791932 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:34.792059 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 12:19:34.792173 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 12:19:34.792336 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260925 12:19:34.792987 14327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32939
I20260925 12:19:34.793087 14327 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:34.793282 14327 heartbeater.cc:511] Master 127.0.98.62:32939 requested a full tablet report, sending...
I20260925 12:19:34.793781 14136 ts_manager.cc:194] Registered new tserver with Master: 18d3e88caaa34418876c3759d893f8fe (127.0.98.1:44365)
I20260925 12:19:34.794339 14136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52011
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:19:34.813855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32939
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:19:34.887322 14332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:34.887509 14332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:34.887560 14332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:34.888643 14332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:34.888713 14332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:34.888756 14332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:19:34.890112 14332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:34.890339 14332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:34.890523 14332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:34.895159 14340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:34.895205 14337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:34.895269 14332 server_base.cc:1034] running on GCE node
W20260925 12:19:34.895220 14338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:34.895576 14332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:34.895860 14332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:34.897007 14332 hybrid_clock.cc:648] HybridClock initialized: now 1790338774896996 us; error 26 us; skew 500 ppm
Sep 25 12:19:34 dist-test-slave-clmx krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:34.899286 14332 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:19:34.899745 14332 webserver.cc:460] Webserver started at http://127.0.98.2:44943/ using document root <none> and password file <none>
I20260925 12:19:34.899950 14332 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:34.900022 14332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:34.900156 14332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:19:34.901111 14332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "2ccb22dea9b04e6fa378297edce10fd5"
format_stamp: "Formatted at 2026-09-25 12:19:34 on dist-test-slave-clmx"
server_key: "3e5a60367d816bf0403ca1929dbd09cd"
server_key_iv: "0198bbca33d89deeb0c43acf55e6ae1c"
server_key_version: "encryptionkey@0"
I20260925 12:19:34.901476 14332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2ccb22dea9b04e6fa378297edce10fd5"
format_stamp: "Formatted at 2026-09-25 12:19:34 on dist-test-slave-clmx"
server_key: "3e5a60367d816bf0403ca1929dbd09cd"
server_key_iv: "0198bbca33d89deeb0c43acf55e6ae1c"
server_key_version: "encryptionkey@0"
I20260925 12:19:34.902688 14332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 12:19:34.903342 14347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:34.903494 14332 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:19:34.903585 14332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "2ccb22dea9b04e6fa378297edce10fd5"
format_stamp: "Formatted at 2026-09-25 12:19:34 on dist-test-slave-clmx"
server_key: "3e5a60367d816bf0403ca1929dbd09cd"
server_key_iv: "0198bbca33d89deeb0c43acf55e6ae1c"
server_key_version: "encryptionkey@0"
I20260925 12:19:34.903666 14332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:34.912135 14332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:34.914312 14332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:34.914450 14332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:34.914644 14332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:19:34.914898 14332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:19:34.914947 14332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:34.914983 14332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:19:34.915009 14332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:34.920370 14332 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46823
I20260925 12:19:34.920455 14460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46823 every 8 connection(s)
I20260925 12:19:34.920740 14332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:19:34 dist-test-slave-clmx krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:34.927773 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 14332
I20260925 12:19:34.927872 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:19:34.928069 392 external_mini_cluster.cc:1428] Setting key 14704a1c57ab41da6a168bb8b79723e7
I20260925 12:19:34.928999 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:34.921391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55745 (local address 127.0.98.62:32939)
0925 12:19:34.921465 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:34.921467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:34.921686 (+ 219us) server_negotiation.cc:407] Connection header received
0925 12:19:34.922135 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:34.922136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:34.922147 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:34.922195 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:34.922977 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:34.923478 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:34.924361 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:34.924536 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:34.924588 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:34.926427 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:34.926428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:34.926428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:34.926434 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:34.927506 (+ 1072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:34.928020 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:34.928021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:34.928021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:34.928068 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:34.928360 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:34.928360 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:34.928361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:34.928476 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:34.928554 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:34.928838 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 12:19:34.928915 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 12:19:34.929153 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:34.921361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:55745)
0925 12:19:34.921530 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:34.921577 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:34.922039 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:34.922240 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:34.922243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:34.922747 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:34.922911 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:34.922914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:34.923549 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:34.923550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:34.924292 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:34.924293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:34.924602 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:34.924602 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:34.924687 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:34.925130 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:34.925138 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:34.926342 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:34.927609 (+ 1267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:34.927615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:34.927617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:34.927936 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:34.928169 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:34.928169 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:34.928170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:34.928281 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:34.928546 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:34.928552 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:34.928709 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 12:19:34.928808 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 12:19:34.928947 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
I20260925 12:19:34.929561 14461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32939
I20260925 12:19:34.929643 14461 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:34.929805 14461 heartbeater.cc:511] Master 127.0.98.62:32939 requested a full tablet report, sending...
I20260925 12:19:34.930183 14136 ts_manager.cc:194] Registered new tserver with Master: 2ccb22dea9b04e6fa378297edce10fd5 (127.0.98.2:46823)
I20260925 12:19:34.930680 14136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55745
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:19:34.950351 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32939
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:19:35.025986 14466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:35.026182 14466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:35.026203 14466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:35.027202 14466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:35.027235 14466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:35.027259 14466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:19:35.028580 14466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:35.028764 14466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:35.028918 14466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:35.033315 14472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:35.033304 14471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:35.033305 14474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:35.033470 14466 server_base.cc:1034] running on GCE node
I20260925 12:19:35.033751 14466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:35.033923 14466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:35.035080 14466 hybrid_clock.cc:648] HybridClock initialized: now 1790338775035068 us; error 27 us; skew 500 ppm
Sep 25 12:19:35 dist-test-slave-clmx krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:35.037393 14466 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:19:35.037834 14466 webserver.cc:460] Webserver started at http://127.0.98.3:34625/ using document root <none> and password file <none>
I20260925 12:19:35.038045 14466 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:35.038107 14466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:35.038249 14466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:19:35.039160 14466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "12d5988fdfc1455bac63cdafe131e287"
format_stamp: "Formatted at 2026-09-25 12:19:35 on dist-test-slave-clmx"
server_key: "7e59f36c9ae20efc03b1b72a40a5cad9"
server_key_iv: "5dbcd933481d77c2feab6fafee078d6a"
server_key_version: "encryptionkey@0"
I20260925 12:19:35.039496 14466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "12d5988fdfc1455bac63cdafe131e287"
format_stamp: "Formatted at 2026-09-25 12:19:35 on dist-test-slave-clmx"
server_key: "7e59f36c9ae20efc03b1b72a40a5cad9"
server_key_iv: "5dbcd933481d77c2feab6fafee078d6a"
server_key_version: "encryptionkey@0"
I20260925 12:19:35.040725 14466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:19:35.041388 14481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:35.041544 14466 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:19:35.041633 14466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "12d5988fdfc1455bac63cdafe131e287"
format_stamp: "Formatted at 2026-09-25 12:19:35 on dist-test-slave-clmx"
server_key: "7e59f36c9ae20efc03b1b72a40a5cad9"
server_key_iv: "5dbcd933481d77c2feab6fafee078d6a"
server_key_version: "encryptionkey@0"
I20260925 12:19:35.041708 14466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:35.054857 14466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:35.056504 14466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:35.056663 14466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:35.056841 14466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:19:35.057106 14466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:19:35.057137 14466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:35.057155 14466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:19:35.057199 14466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:35.062577 14466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40589
I20260925 12:19:35.062672 14594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40589 every 8 connection(s)
I20260925 12:19:35.062924 14466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:19:35.064129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 14466
I20260925 12:19:35.064224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:19:35.064393 392 external_mini_cluster.cc:1428] Setting key 5473d946b0c824d6299b9d006a8fe0f3
Sep 25 12:19:35 dist-test-slave-clmx krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:35.071346 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.063551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41523 (local address 127.0.98.62:32939)
0925 12:19:35.063670 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:35.063673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:35.063827 (+ 154us) server_negotiation.cc:407] Connection header received
0925 12:19:35.064492 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:35.064493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:35.064505 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:35.064569 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:35.065586 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.066041 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.066902 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.067078 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.067124 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.069070 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:35.069076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:35.069076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:35.069082 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:35.070360 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:35.070668 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:35.070669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:35.070687 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:35.070747 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:35.070925 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:35.070925 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:35.070925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:35.071008 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:35.071051 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:35.071241 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 12:19:35.071291 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 12:19:35.071504 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.063538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:41523)
0925 12:19:35.063714 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:35.063751 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:35.064381 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:35.064653 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:35.064657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:35.065338 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:35.065505 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.065509 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.066112 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.066113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.066831 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.066833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.067142 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.067143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.067233 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.067720 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:35.067730 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:35.068985 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:35.070421 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:35.070426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:35.070427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:35.070611 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:35.070797 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:35.070798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:35.070798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:35.070877 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:35.071058 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:35.071061 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:35.071150 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 12:19:35.071218 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 12:19:35.071313 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260925 12:19:35.071889 14595 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32939
I20260925 12:19:35.071974 14595 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:35.072146 14595 heartbeater.cc:511] Master 127.0.98.62:32939 requested a full tablet report, sending...
I20260925 12:19:35.072470 14136 ts_manager.cc:194] Registered new tserver with Master: 12d5988fdfc1455bac63cdafe131e287 (127.0.98.3:40589)
I20260925 12:19:35.072798 14136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41523
I20260925 12:19:35.075825 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:19:35.081990 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.076972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.0.98.62:32939)
0925 12:19:35.077018 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:35.077020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:35.077079 (+ 59us) server_negotiation.cc:407] Connection header received
0925 12:19:35.077129 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:35.077130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:35.077137 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:35.077183 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:35.077643 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.078056 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.078684 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.078842 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.078885 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.079913 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:35.079914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:35.079914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:35.079919 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:35.081216 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:35.081474 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:35.081474 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:35.081474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:35.081517 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:35.081637 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:35.081637 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:35.081638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:35.081742 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:35.081807 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:35.081893 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 12:19:35.081940 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 12:19:35.083554 14136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:19:35.085143 14136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:19:35.090795 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.087041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40589 (local address 127.0.0.1:58748)
0925 12:19:35.087408 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:35.087418 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:35.087470 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:35.087678 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:35.087678 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:35.087679 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:19:35.087724 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.087725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.088676 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.088677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.089831 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.089833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.090488 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.090489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.090572 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.090584 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 12:19:35.090679 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 12:19:35.090729 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":15,"thread_start_us":89,"threads_started":1}
I20260925 12:19:35.090802 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.087234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58748 (local address 127.0.98.3:40589)
0925 12:19:35.087515 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:35.087518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:35.087526 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:19:35.087556 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:35.087557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:35.087570 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:35.087671 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:19:35.087802 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.088569 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.089920 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.090405 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.090487 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.090538 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:35.090696 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 12:19:35.090729 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":71,"threads_started":1}
I20260925 12:19:35.091079 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.087020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46823 (local address 127.0.0.1:42574)
0925 12:19:35.087358 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:35.087367 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:35.087403 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:35.087657 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:35.087659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:35.087661 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:19:35.087706 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.087708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.088989 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.088989 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.089986 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.089987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.090804 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.090805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.090860 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.090862 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:19:35.090904 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 12:19:35.090933 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":81,"threads_started":1}
I20260925 12:19:35.091166 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.087247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42574 (local address 127.0.98.2:46823)
0925 12:19:35.087400 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:35.087404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:35.087416 (+ 12us) server_negotiation.cc:407] Connection header received
0925 12:19:35.087458 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:35.087460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:35.087475 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:35.087546 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:19:35.087774 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.088873 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.090046 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.090672 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.090729 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.090779 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:35.091034 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 12:19:35.091085 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260925 12:19:35.090970 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.087013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44365 (local address 127.0.0.1:40546)
0925 12:19:35.087171 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:35.087204 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:35.087264 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:35.087600 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:35.087603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:35.087606 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:19:35.087678 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.087680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.089051 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.089052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.089872 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.089875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.090756 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.090757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.090819 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.090822 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:19:35.090851 (+ 29us) client_negotiation.cc:240] Negotiation successful
0925 12:19:35.090894 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":56,"threads_started":1}
I20260925 12:19:35.091409 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.087198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40546 (local address 127.0.98.1:44365)
0925 12:19:35.087383 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:35.087388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:35.087397 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:19:35.087434 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:35.087436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:35.087450 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:35.087520 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:19:35.087811 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.088943 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.089973 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.090655 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.091055 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.091126 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:35.091147 (+ 21us) server_negotiation.cc:299] Negotiation successful
0925 12:19:35.091332 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
I20260925 12:19:35.091754 14529 tablet_service.cc:1467] Processing CreateTablet for tablet f7d1b005fe4a4e3cb4e55897d8f88af1 (DEFAULT_TABLE table=test-table [id=0d577f4f571145339203e4d4515ff1fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:19:35.091827 14395 tablet_service.cc:1467] Processing CreateTablet for tablet f7d1b005fe4a4e3cb4e55897d8f88af1 (DEFAULT_TABLE table=test-table [id=0d577f4f571145339203e4d4515ff1fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:19:35.092087 14529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d1b005fe4a4e3cb4e55897d8f88af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:19:35.092123 14395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d1b005fe4a4e3cb4e55897d8f88af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:19:35.092039 14261 tablet_service.cc:1467] Processing CreateTablet for tablet f7d1b005fe4a4e3cb4e55897d8f88af1 (DEFAULT_TABLE table=test-table [id=0d577f4f571145339203e4d4515ff1fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:19:35.092424 14261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d1b005fe4a4e3cb4e55897d8f88af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:19:35.094681 14609 tablet_bootstrap.cc:492] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe: Bootstrap starting.
I20260925 12:19:35.095181 14611 tablet_bootstrap.cc:492] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287: Bootstrap starting.
I20260925 12:19:35.095377 14609 tablet_bootstrap.cc:654] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe: Neither blocks nor log segments found. Creating new log.
I20260925 12:19:35.095589 14610 tablet_bootstrap.cc:492] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5: Bootstrap starting.
I20260925 12:19:35.095947 14611 tablet_bootstrap.cc:654] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287: Neither blocks nor log segments found. Creating new log.
I20260925 12:19:35.095959 14609 log.cc:824] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:35.096210 14610 tablet_bootstrap.cc:654] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5: Neither blocks nor log segments found. Creating new log.
I20260925 12:19:35.096465 14611 log.cc:824] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:35.096753 14610 log.cc:824] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:35.096868 14609 tablet_bootstrap.cc:492] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe: No bootstrap required, opened a new log
I20260925 12:19:35.096938 14609 ts_tablet_manager.cc:1397] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:19:35.097134 14611 tablet_bootstrap.cc:492] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287: No bootstrap required, opened a new log
I20260925 12:19:35.097211 14611 ts_tablet_manager.cc:1397] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 12:19:35.098448 14611 raft_consensus.cc:348] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } }
I20260925 12:19:35.098603 14611 raft_consensus.cc:374] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:19:35.098649 14611 raft_consensus.cc:729] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d5988fdfc1455bac63cdafe131e287, State: Initialized, Role: FOLLOWER
I20260925 12:19:35.098531 14609 raft_consensus.cc:348] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } }
I20260925 12:19:35.098687 14609 raft_consensus.cc:374] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:19:35.098745 14609 raft_consensus.cc:729] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d3e88caaa34418876c3759d893f8fe, State: Initialized, Role: FOLLOWER
I20260925 12:19:35.098762 14611 consensus_queue.cc:260] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } }
I20260925 12:19:35.098909 14609 consensus_queue.cc:260] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } }
I20260925 12:19:35.099026 14611 ts_tablet_manager.cc:1428] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 12:19:35.099085 14595 heartbeater.cc:503] Master 127.0.98.62:32939 was elected leader, sending a full tablet report...
I20260925 12:19:35.099171 14609 ts_tablet_manager.cc:1428] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:19:35.099188 14327 heartbeater.cc:503] Master 127.0.98.62:32939 was elected leader, sending a full tablet report...
I20260925 12:19:35.099757 14610 tablet_bootstrap.cc:492] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5: No bootstrap required, opened a new log
I20260925 12:19:35.099860 14610 ts_tablet_manager.cc:1397] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 12:19:35.101192 14610 raft_consensus.cc:348] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } }
I20260925 12:19:35.101306 14610 raft_consensus.cc:374] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:19:35.101326 14610 raft_consensus.cc:729] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ccb22dea9b04e6fa378297edce10fd5, State: Initialized, Role: FOLLOWER
I20260925 12:19:35.101438 14610 consensus_queue.cc:260] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } }
I20260925 12:19:35.101703 14610 ts_tablet_manager.cc:1428] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:19:35.101725 14461 heartbeater.cc:503] Master 127.0.98.62:32939 was elected leader, sending a full tablet report...
W20260925 12:19:35.171244 14462 tablet.cc:2369] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:19:35.179253 14615 raft_consensus.cc:482] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:19:35.179368 14615 raft_consensus.cc:504] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } }
I20260925 12:19:35.179746 14615 leader_election.cc:288] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18d3e88caaa34418876c3759d893f8fe (127.0.98.1:44365), 2ccb22dea9b04e6fa378297edce10fd5 (127.0.98.2:46823)
I20260925 12:19:35.183472 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.179986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57823 (local address 127.0.98.2:46823)
0925 12:19:35.180035 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:35.180037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:35.180179 (+ 142us) server_negotiation.cc:407] Connection header received
0925 12:19:35.180203 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:35.180203 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:35.180212 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:35.180253 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:19:35.180406 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.181082 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.182467 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.183094 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.183151 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.183158 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:35.183363 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 12:19:35.183406 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 12:19:35.183486 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.179873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44365 (local address 127.0.98.3:40865)
0925 12:19:35.179922 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:35.179934 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:35.179975 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:35.180144 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:35.180145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:35.180147 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:19:35.180218 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.180220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.181185 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.181186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.182283 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.182286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.183166 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.183168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.183278 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.183283 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:19:35.183345 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 12:19:35.183389 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 12:19:35.183499 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.179895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46823 (local address 127.0.98.3:57823)
0925 12:19:35.180094 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:35.180104 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:35.180153 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:35.180275 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:35.180276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:35.180277 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:19:35.180336 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.180337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.181196 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.181198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.182391 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.182394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.183204 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.183205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.183282 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.183286 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:19:35.183336 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 12:19:35.183374 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":109,"threads_started":1}
I20260925 12:19:35.183898 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.179927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40865 (local address 127.0.98.1:44365)
0925 12:19:35.179967 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:35.179969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:35.179978 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:19:35.180033 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:35.180034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:35.180046 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:35.180115 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:19:35.180315 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.181089 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.182350 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.183069 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.183147 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.183158 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:35.183775 (+ 617us) server_negotiation.cc:299] Negotiation successful
0925 12:19:35.183821 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:19:35.184171 14415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d1b005fe4a4e3cb4e55897d8f88af1" candidate_uuid: "12d5988fdfc1455bac63cdafe131e287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ccb22dea9b04e6fa378297edce10fd5" is_pre_election: true
I20260925 12:19:35.184188 14281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d1b005fe4a4e3cb4e55897d8f88af1" candidate_uuid: "12d5988fdfc1455bac63cdafe131e287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d3e88caaa34418876c3759d893f8fe" is_pre_election: true
I20260925 12:19:35.184311 14415 raft_consensus.cc:2454] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12d5988fdfc1455bac63cdafe131e287 in term 0.
I20260925 12:19:35.184331 14281 raft_consensus.cc:2454] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12d5988fdfc1455bac63cdafe131e287 in term 0.
I20260925 12:19:35.184525 14485 leader_election.cc:302] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d5988fdfc1455bac63cdafe131e287, 2ccb22dea9b04e6fa378297edce10fd5; no voters:
I20260925 12:19:35.184686 14615 raft_consensus.cc:2781] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:19:35.184762 14615 raft_consensus.cc:482] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:19:35.184798 14615 raft_consensus.cc:3037] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:19:35.185472 14615 raft_consensus.cc:504] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } }
I20260925 12:19:35.185617 14615 leader_election.cc:288] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [CANDIDATE]: Term 1 election: Requested vote from peers 18d3e88caaa34418876c3759d893f8fe (127.0.98.1:44365), 2ccb22dea9b04e6fa378297edce10fd5 (127.0.98.2:46823)
I20260925 12:19:35.185740 14281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d1b005fe4a4e3cb4e55897d8f88af1" candidate_uuid: "12d5988fdfc1455bac63cdafe131e287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d3e88caaa34418876c3759d893f8fe"
I20260925 12:19:35.185778 14415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d1b005fe4a4e3cb4e55897d8f88af1" candidate_uuid: "12d5988fdfc1455bac63cdafe131e287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ccb22dea9b04e6fa378297edce10fd5"
I20260925 12:19:35.185818 14281 raft_consensus.cc:3037] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:19:35.185834 14415 raft_consensus.cc:3037] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:19:35.186673 14281 raft_consensus.cc:2454] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12d5988fdfc1455bac63cdafe131e287 in term 1.
I20260925 12:19:35.186673 14415 raft_consensus.cc:2454] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12d5988fdfc1455bac63cdafe131e287 in term 1.
I20260925 12:19:35.186815 14485 leader_election.cc:302] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d5988fdfc1455bac63cdafe131e287, 2ccb22dea9b04e6fa378297edce10fd5; no voters:
I20260925 12:19:35.186897 14615 raft_consensus.cc:2781] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:19:35.187032 14615 raft_consensus.cc:686] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [term 1 LEADER]: Becoming Leader. State: Replica: 12d5988fdfc1455bac63cdafe131e287, State: Running, Role: LEADER
I20260925 12:19:35.187134 14615 consensus_queue.cc:237] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } }
I20260925 12:19:35.187875 14135 catalog_manager.cc:5495] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12d5988fdfc1455bac63cdafe131e287 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "12d5988fdfc1455bac63cdafe131e287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } health_report { overall_health: HEALTHY } } }
I20260925 12:19:35.209658 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.207495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58762 (local address 127.0.98.3:40589)
0925 12:19:35.207560 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:35.207561 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:35.207568 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:19:35.207598 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:35.207599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:35.207662 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:35.207735 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:19:35.207844 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.208254 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.208961 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.209147 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.209217 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.209375 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:19:35.209465 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:19:35.209555 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:35.209573 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 12:19:35.209592 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 12:19:35.211150 14281 raft_consensus.cc:1261] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe [term 1 FOLLOWER]: Refusing update from remote peer 12d5988fdfc1455bac63cdafe131e287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:19:35.211150 14415 raft_consensus.cc:1261] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5 [term 1 FOLLOWER]: Refusing update from remote peer 12d5988fdfc1455bac63cdafe131e287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:19:35.211498 14619 consensus_queue.cc:1035] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:19:35.211653 14615 consensus_queue.cc:1035] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:19:35.213673 14622 mvcc.cc:205] Tried to move back new op lower bound from 7333227623262605312 to 7333227623167152128. Current Snapshot: MvccSnapshot[applied={T|T < 7333227623262605312}]
I20260925 12:19:35.216074 14136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:19:35.216166 14136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:19:35.216883 14136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a: Sending DeleteTablet for 3 replicas of tablet f7d1b005fe4a4e3cb4e55897d8f88af1
I20260925 12:19:35.217214 14395 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d1b005fe4a4e3cb4e55897d8f88af1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:19:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42574
I20260925 12:19:35.217259 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 14198
I20260925 12:19:35.217406 14261 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d1b005fe4a4e3cb4e55897d8f88af1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:19:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40546
I20260925 12:19:35.217967 14631 tablet_replica.cc:330] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5: stopping tablet replica
I20260925 12:19:35.217962 14529 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d1b005fe4a4e3cb4e55897d8f88af1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:19:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58748
I20260925 12:19:35.218065 14631 raft_consensus.cc:2229] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:19:35.218143 14632 tablet_replica.cc:330] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287: stopping tablet replica
I20260925 12:19:35.218150 14631 raft_consensus.cc:2258] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:19:35.218219 14632 raft_consensus.cc:2229] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:19:35.218292 14632 raft_consensus.cc:2258] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:19:35.218544 14631 ts_tablet_manager.cc:1905] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:19:35.218688 14632 ts_tablet_manager.cc:1905] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:19:35.220513 14631 ts_tablet_manager.cc:1918] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:19:35.220528 14632 ts_tablet_manager.cc:1918] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:19:35.220585 14631 log.cc:1196] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/f7d1b005fe4a4e3cb4e55897d8f88af1
I20260925 12:19:35.220588 14632 log.cc:1196] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/f7d1b005fe4a4e3cb4e55897d8f88af1
I20260925 12:19:35.220870 14631 ts_tablet_manager.cc:1939] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 2ccb22dea9b04e6fa378297edce10fd5: Deleting consensus metadata
I20260925 12:19:35.220894 14632 ts_tablet_manager.cc:1939] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 12d5988fdfc1455bac63cdafe131e287: Deleting consensus metadata
I20260925 12:19:35.221293 14120 catalog_manager.cc:4841] TS 12d5988fdfc1455bac63cdafe131e287 (127.0.98.3:40589): tablet f7d1b005fe4a4e3cb4e55897d8f88af1 (table test-table [id=0d577f4f571145339203e4d4515ff1fb]) successfully deleted
I20260925 12:19:35.221308 14123 catalog_manager.cc:4841] TS 2ccb22dea9b04e6fa378297edce10fd5 (127.0.98.2:46823): tablet f7d1b005fe4a4e3cb4e55897d8f88af1 (table test-table [id=0d577f4f571145339203e4d4515ff1fb]) successfully deleted
I20260925 12:19:35.224110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 14332
W20260925 12:19:35.224217 14122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44365 (error 108)
I20260925 12:19:35.225255 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.224804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44365 (local address 127.0.0.1:40562)
0925 12:19:35.225172 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:35.225218 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347}
W20260925 12:19:35.225395 14122 catalog_manager.cc:4584] TS 18d3e88caaa34418876c3759d893f8fe (127.0.98.1:44365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7d1b005fe4a4e3cb4e55897d8f88af1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44365: connect: Connection refused (error 111)
I20260925 12:19:35.229810 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 14466
I20260925 12:19:35.234205 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 14104
I20260925 12:19:35.239320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32939
--webserver_interface=127.0.98.62
--webserver_port=34387
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:19:35.316758 14634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:35.316933 14634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:35.316954 14634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:35.318059 14634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:19:35.318099 14634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:35.318112 14634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:35.318125 14634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:19:35.318177 14634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:19:35.319548 14634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:35.319811 14634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:35.320037 14634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:35.328150 14640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:35.328145 14641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:35.328145 14643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:35.328500 14634 server_base.cc:1034] running on GCE node
I20260925 12:19:35.328770 14634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:35.329056 14634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:35.330245 14634 hybrid_clock.cc:648] HybridClock initialized: now 1790338775330233 us; error 46 us; skew 500 ppm
Sep 25 12:19:35 dist-test-slave-clmx krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:35.332736 14634 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:19:35.333156 14634 webserver.cc:460] Webserver started at http://127.0.98.62:34387/ using document root <none> and password file <none>
I20260925 12:19:35.333423 14634 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:35.333500 14634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:35.334789 14634 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:19:35.335366 14650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:35.335515 14634 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:19:35.335615 14634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "43c4b6e343b64336b021ab9c29c13f3a"
format_stamp: "Formatted at 2026-09-25 12:19:34 on dist-test-slave-clmx"
server_key: "c1178a865042d05fb76af66a379c8484"
server_key_iv: "eb10a6dd3d57b448e4e3501071f1335c"
server_key_version: "encryptionkey@0"
I20260925 12:19:35.335984 14634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:35.351003 14634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:35.353231 14634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:35.353403 14634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:35.357139 14634 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32939
I20260925 12:19:35.357167 14702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32939 every 8 connection(s)
I20260925 12:19:35.357517 14634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:19:35.359005 14703 sys_catalog.cc:263] Verifying existing consensus state
I20260925 12:19:35.359736 14703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a: Bootstrap starting.
I20260925 12:19:35.363644 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 14634
I20260925 12:19:35.363955 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44365
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:32939
--webserver_port=32897
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:19:35.368075 14703 log.cc:824] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:35.369767 14703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:19:35.370030 14703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a: Bootstrap complete.
I20260925 12:19:35.372216 14703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4b6e343b64336b021ab9c29c13f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } }
I20260925 12:19:35.372426 14703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c4b6e343b64336b021ab9c29c13f3a, State: Initialized, Role: FOLLOWER
I20260925 12:19:35.372589 14703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4b6e343b64336b021ab9c29c13f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } }
I20260925 12:19:35.372648 14703 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:19:35.372713 14703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:19:35.372766 14703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 1 FOLLOWER]: Advancing to term 2
I20260925 12:19:35.373718 14703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4b6e343b64336b021ab9c29c13f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } }
I20260925 12:19:35.373799 14703 leader_election.cc:302] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43c4b6e343b64336b021ab9c29c13f3a; no voters:
I20260925 12:19:35.374007 14703 leader_election.cc:288] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 12:19:35.374040 14707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 2 FOLLOWER]: Leader election won for term 2
I20260925 12:19:35.374146 14707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 2 LEADER]: Becoming Leader. State: Replica: 43c4b6e343b64336b021ab9c29c13f3a, State: Running, Role: LEADER
I20260925 12:19:35.374239 14707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4b6e343b64336b021ab9c29c13f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } }
I20260925 12:19:35.374266 14703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:19:35.374512 14708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "43c4b6e343b64336b021ab9c29c13f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4b6e343b64336b021ab9c29c13f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } } }
I20260925 12:19:35.374667 14708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [sys.catalog]: This master's current role is: LEADER
I20260925 12:19:35.374926 14712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:19:35.375712 14712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:19:35.375751 14709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43c4b6e343b64336b021ab9c29c13f3a. Latest consensus state: current_term: 2 leader_uuid: "43c4b6e343b64336b021ab9c29c13f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4b6e343b64336b021ab9c29c13f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } } }
I20260925 12:19:35.375811 14709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [sys.catalog]: This master's current role is: LEADER
I20260925 12:19:35.375816 14712 catalog_manager.cc:1259] Loaded cluster ID: 6ea7f03a141b4dd9a9d6a3518892a723
I20260925 12:19:35.375918 14712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:19:35.376129 14712 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 12:19:35.378969 14712 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 12:19:35.384661 14712 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:19:35.384861 14712 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a: Loaded TSK: 0
I20260925 12:19:35.385068 14712 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 12:19:35.460666 14705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:35.460862 14705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:35.460883 14705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:35.462831 14705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:35.462889 14705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:35.462935 14705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:19:35.465524 14705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=32897
--tserver_master_addrs=127.0.98.62:32939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:35.465751 14705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:35.466006 14705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:35.472350 14733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:35.472338 14731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:35.472469 14730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:35.472728 14705 server_base.cc:1034] running on GCE node
I20260925 12:19:35.472910 14705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:35.473188 14705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:35.474337 14705 hybrid_clock.cc:648] HybridClock initialized: now 1790338775474327 us; error 30 us; skew 500 ppm
Sep 25 12:19:35 dist-test-slave-clmx krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:35.477056 14705 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:19:35.477473 14705 webserver.cc:460] Webserver started at http://127.0.98.1:32897/ using document root <none> and password file <none>
I20260925 12:19:35.477669 14705 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:35.477739 14705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:35.479005 14705 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:19:35.479674 14740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:35.479873 14705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:19:35.479948 14705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "18d3e88caaa34418876c3759d893f8fe"
format_stamp: "Formatted at 2026-09-25 12:19:34 on dist-test-slave-clmx"
server_key: "a1272ee60da76e59fdec3a0e9838de01"
server_key_iv: "ccf7711c75097f1f0057c4e4d3a249f1"
server_key_version: "encryptionkey@0"
I20260925 12:19:35.480240 14705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:35.491259 14705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:35.493150 14705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:35.493291 14705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:35.493541 14705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:19:35.493947 14747 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 12:19:35.495667 14705 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 12:19:35.495724 14705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 12:19:35.495759 14705 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 12:19:35.496263 14705 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 12:19:35.496313 14705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 12:19:35.496377 14747 tablet_bootstrap.cc:492] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe: Bootstrap starting.
I20260925 12:19:35.503036 14705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44365
I20260925 12:19:35.503064 14854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44365 every 8 connection(s)
I20260925 12:19:35.503450 14705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:19:35.508822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 14705
I20260925 12:19:35.509145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46823
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:32939
--webserver_port=44943
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:19:35.509935 14747 log.cc:824] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:35.512681 14747 tablet_bootstrap.cc:492] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:19:35.512991 14747 tablet_bootstrap.cc:492] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe: Bootstrap complete.
Sep 25 12:19:35 dist-test-slave-clmx krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:35.514231 14747 ts_tablet_manager.cc:1397] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.008s
I20260925 12:19:35.515872 14747 raft_consensus.cc:348] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } }
I20260925 12:19:35.516245 14747 raft_consensus.cc:729] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d3e88caaa34418876c3759d893f8fe, State: Initialized, Role: FOLLOWER
I20260925 12:19:35.516407 14747 consensus_queue.cc:260] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } }
I20260925 12:19:35.516705 14747 ts_tablet_manager.cc:1428] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260925 12:19:35.518208 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.504182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33391 (local address 127.0.98.62:32939)
0925 12:19:35.504396 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:35.504404 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:35.504438 (+ 34us) server_negotiation.cc:407] Connection header received
0925 12:19:35.505111 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:35.505118 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:35.505193 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:35.505360 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:35.506107 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.507509 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.508978 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.509554 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.509655 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.514061 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:35.514065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:35.514066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:35.514077 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:35.516112 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:35.516795 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:35.516798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:35.516798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:35.516895 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:35.517172 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:35.517173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:35.517173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:35.517425 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:35.517503 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:35.517870 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 12:19:35.517996 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":63,"threads_started":1}
I20260925 12:19:35.518209 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.504092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:33391)
0925 12:19:35.504281 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:35.504324 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:35.504845 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:35.505403 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:35.505408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:35.505718 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:35.505960 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.505965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.507661 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.507662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.508880 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.508883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.509769 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.509770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.509888 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.510401 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:35.510410 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:35.513925 (+ 3515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:35.516202 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:35.516207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:35.516209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:35.516674 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:35.516954 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:35.516955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:35.516955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:35.517087 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:35.517506 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:35.517513 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:35.517685 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 12:19:35.517807 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 12:19:35.517979 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":98,"threads_started":1}
I20260925 12:19:35.519708 14855 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32939
I20260925 12:19:35.519804 14855 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:35.519980 14855 heartbeater.cc:511] Master 127.0.98.62:32939 requested a full tablet report, sending...
I20260925 12:19:35.520736 14667 ts_manager.cc:194] Registered new tserver with Master: 18d3e88caaa34418876c3759d893f8fe (127.0.98.1:44365)
I20260925 12:19:35.521867 14667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33391
I20260925 12:19:35.522917 14855 heartbeater.cc:503] Master 127.0.98.62:32939 was elected leader, sending a full tablet report...
I20260925 12:19:35.527706 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.521509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.0.98.1:44365)
0925 12:19:35.523743 (+ 2234us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:35.523747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:35.523802 (+ 55us) server_negotiation.cc:407] Connection header received
0925 12:19:35.523890 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:35.523892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:35.523906 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:35.523992 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:19:35.524127 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.525210 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.526674 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.527338 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.527409 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.527453 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:35.527595 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 12:19:35.527643 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2205,"thread_start_us":55,"threads_started":1}
I20260925 12:19:35.527710 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.521303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44365 (local address 127.0.0.1:40566)
0925 12:19:35.523718 (+ 2415us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:35.523752 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:35.523819 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:35.524001 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:35.524004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:35.524006 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:19:35.524059 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.524060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.525294 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.525295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.526563 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.526566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.527418 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.527419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.527496 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.527508 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 12:19:35.527573 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 12:19:35.527641 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2387,"thread_start_us":73,"threads_started":1}
I20260925 12:19:35.528477 14789 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d1b005fe4a4e3cb4e55897d8f88af1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:19:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40566
I20260925 12:19:35.529007 14864 tablet_replica.cc:330] stopping tablet replica
I20260925 12:19:35.529102 14864 raft_consensus.cc:2229] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:19:35.529156 14864 raft_consensus.cc:2258] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:19:35.529412 14864 ts_tablet_manager.cc:1905] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:19:35.531692 14864 ts_tablet_manager.cc:1918] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:19:35.531760 14864 log.cc:1196] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/wal/wals/f7d1b005fe4a4e3cb4e55897d8f88af1
I20260925 12:19:35.531966 14864 ts_tablet_manager.cc:1939] T f7d1b005fe4a4e3cb4e55897d8f88af1 P 18d3e88caaa34418876c3759d893f8fe: Deleting consensus metadata
I20260925 12:19:35.532433 14653 catalog_manager.cc:4841] TS 18d3e88caaa34418876c3759d893f8fe (127.0.98.1:44365): tablet f7d1b005fe4a4e3cb4e55897d8f88af1 (table test-table [id=0d577f4f571145339203e4d4515ff1fb]) successfully deleted
W20260925 12:19:35.592041 14860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:35.592260 14860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:35.592296 14860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:35.593420 14860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:35.593473 14860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:35.593513 14860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:19:35.594818 14860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44943
--tserver_master_addrs=127.0.98.62:32939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:35.595039 14860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:35.595225 14860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:35.599666 14872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:35.599644 14870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:35.599720 14869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:35.599815 14860 server_base.cc:1034] running on GCE node
I20260925 12:19:35.599980 14860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:35.600188 14860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:35.601325 14860 hybrid_clock.cc:648] HybridClock initialized: now 1790338775601316 us; error 25 us; skew 500 ppm
Sep 25 12:19:35 dist-test-slave-clmx krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:35.603518 14860 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:19:35.603928 14860 webserver.cc:460] Webserver started at http://127.0.98.2:44943/ using document root <none> and password file <none>
I20260925 12:19:35.604136 14860 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:35.604214 14860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:35.605432 14860 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:19:35.605975 14879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:35.606138 14860 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:19:35.606227 14860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "2ccb22dea9b04e6fa378297edce10fd5"
format_stamp: "Formatted at 2026-09-25 12:19:34 on dist-test-slave-clmx"
server_key: "3e5a60367d816bf0403ca1929dbd09cd"
server_key_iv: "0198bbca33d89deeb0c43acf55e6ae1c"
server_key_version: "encryptionkey@0"
I20260925 12:19:35.606528 14860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:35.626571 14860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:35.628185 14860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:35.628332 14860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:35.628532 14860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:19:35.628803 14860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:19:35.628854 14860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:35.628888 14860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:19:35.628916 14860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:35.634570 14860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46823
I20260925 12:19:35.634604 14992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46823 every 8 connection(s)
I20260925 12:19:35.634897 14860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:19:35 dist-test-slave-clmx krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:35.643442 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.635488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52009 (local address 127.0.98.62:32939)
0925 12:19:35.635530 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:35.635533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:35.635738 (+ 205us) server_negotiation.cc:407] Connection header received
0925 12:19:35.636217 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:35.636218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:35.636232 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:35.636281 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:35.637310 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.637874 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.638803 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.639018 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.639114 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.640981 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:35.640983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:35.640983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:35.640990 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:35.642089 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:35.642504 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:35.642505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:35.642506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:35.642567 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:35.642805 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:35.642806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:35.642806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:35.642920 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:35.642990 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:35.643274 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 12:19:35.643356 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 12:19:35.643605 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.635480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:52009)
0925 12:19:35.635604 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:35.635656 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:35.636102 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:35.636335 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:35.636341 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:35.637067 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:35.637222 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.637226 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.637956 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.637957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.638706 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.638708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.639123 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.639124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.639246 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.639580 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:35.639587 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:35.640894 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:35.642203 (+ 1309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:35.642210 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:35.642211 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:35.642414 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:35.642634 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:35.642634 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:35.642635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:35.642735 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:35.642993 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:35.642999 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:35.643121 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 12:19:35.643250 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 12:19:35.643388 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":71,"threads_started":1}
I20260925 12:19:35.644049 14993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32939
I20260925 12:19:35.644117 14993 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:35.644230 14993 heartbeater.cc:511] Master 127.0.98.62:32939 requested a full tablet report, sending...
I20260925 12:19:35.644515 14667 ts_manager.cc:194] Registered new tserver with Master: 2ccb22dea9b04e6fa378297edce10fd5 (127.0.98.2:46823)
I20260925 12:19:35.644649 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 14860
I20260925 12:19:35.644915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40589
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:32939
--webserver_port=34625
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:19:35.644932 14667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52009
W20260925 12:19:35.721516 14996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:35.721697 14996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:35.721719 14996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:35.722837 14996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:35.722877 14996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:35.722929 14996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:19:35.724422 14996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34625
--tserver_master_addrs=127.0.98.62:32939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:35.724646 14996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:35.724845 14996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:35.729488 15004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:35.729535 14996 server_base.cc:1034] running on GCE node
W20260925 12:19:35.729504 15002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:35.729499 15001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:35.729803 14996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:35.729975 14996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:35.731119 14996 hybrid_clock.cc:648] HybridClock initialized: now 1790338775731110 us; error 23 us; skew 500 ppm
Sep 25 12:19:35 dist-test-slave-clmx krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:35.733486 14996 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:19:35.733901 14996 webserver.cc:460] Webserver started at http://127.0.98.3:34625/ using document root <none> and password file <none>
I20260925 12:19:35.734128 14996 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:35.734206 14996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:35.735337 14996 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:19:35.735947 15011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:35.736125 14996 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:35.736213 14996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "12d5988fdfc1455bac63cdafe131e287"
format_stamp: "Formatted at 2026-09-25 12:19:35 on dist-test-slave-clmx"
server_key: "7e59f36c9ae20efc03b1b72a40a5cad9"
server_key_iv: "5dbcd933481d77c2feab6fafee078d6a"
server_key_version: "encryptionkey@0"
I20260925 12:19:35.736488 14996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:35.758198 14996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:35.759815 14996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:35.759963 14996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:35.760161 14996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:19:35.760433 14996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:19:35.760490 14996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:35.760521 14996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:19:35.760555 14996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:35.765805 14996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40589
I20260925 12:19:35.765843 15124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40589 every 8 connection(s)
I20260925 12:19:35.766141 14996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:19:35.769277 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 14996
Sep 25 12:19:35 dist-test-slave-clmx krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:35.775492 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.766744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45727 (local address 127.0.98.62:32939)
0925 12:19:35.766826 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:35.766828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:35.767014 (+ 186us) server_negotiation.cc:407] Connection header received
0925 12:19:35.767458 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:35.767459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:35.767471 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:35.767521 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:35.768429 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.768922 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.770174 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.770360 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.770439 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.772524 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:35.772526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:35.772526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:35.772534 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:35.774081 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:35.774510 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:35.774511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:35.774511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:35.774565 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:35.774842 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:35.774842 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:35.774843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:35.774961 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:35.775026 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:35.775324 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 12:19:35.775396 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 12:19:35.775678 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.766737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:45727)
0925 12:19:35.766902 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:35.766938 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:35.767359 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:35.767592 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:35.767596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:35.768173 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:35.768334 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.768338 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.769019 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.769020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.770091 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.770093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.770448 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.770449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.770575 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.771002 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:35.771009 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:35.772417 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:35.774163 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:35.774169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:35.774171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:35.774440 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:35.774628 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:35.774629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:35.774630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:35.774770 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:35.775024 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:35.775028 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:35.775145 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 12:19:35.775279 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 12:19:35.775426 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260925 12:19:35.776118 15125 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32939
I20260925 12:19:35.776212 15125 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:35.776371 15125 heartbeater.cc:511] Master 127.0.98.62:32939 requested a full tablet report, sending...
I20260925 12:19:35.776427 15129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.769619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60652 (local address 127.0.98.62:32939)
0925 12:19:35.769730 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:35.769732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:35.769801 (+ 69us) server_negotiation.cc:407] Connection header received
0925 12:19:35.769848 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:35.769849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:35.769859 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:35.769922 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:35.770424 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.771020 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.771795 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.771993 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.772065 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.773168 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:35.773169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:35.773169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:35.773176 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:35.774762 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:35.775183 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:35.775184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:35.775185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:35.775242 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:35.775442 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:35.775443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:35.775443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:35.775578 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:35.775783 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:35.776225 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 12:19:35.776303 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":48,"threads_started":1}
I20260925 12:19:35.776736 14667 ts_manager.cc:194] Registered new tserver with Master: 12d5988fdfc1455bac63cdafe131e287 (127.0.98.3:40589)
I20260925 12:19:35.777055 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:19:35.777083 14667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45727
I20260925 12:19:35.779428 15129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.777369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60660 (local address 127.0.98.62:32939)
0925 12:19:35.777717 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:35.777719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:35.777726 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:19:35.777748 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:35.777749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:35.777770 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:35.777824 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:19:35.777897 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.778276 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.778911 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.779087 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.779130 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.779238 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:19:35.779307 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:19:35.779352 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:35.779381 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 12:19:35.779393 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334}
I20260925 12:19:35.779680 14667 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:19:35.781100 14667 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:19:35.783113 14789 tablet_service.cc:1467] Processing CreateTablet for tablet 6b84ac78aca74f1d92b89e438b1b8b13 (DEFAULT_TABLE table=test-table [id=0348e1bdec044ee48d2d0800fe4a62ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:19:35.783300 14789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b84ac78aca74f1d92b89e438b1b8b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:19:35.784870 14747 tablet_bootstrap.cc:492] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe: Bootstrap starting.
I20260925 12:19:35.785207 14747 tablet_bootstrap.cc:654] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe: Neither blocks nor log segments found. Creating new log.
I20260925 12:19:35.785714 14747 tablet_bootstrap.cc:492] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe: No bootstrap required, opened a new log
I20260925 12:19:35.785770 14747 ts_tablet_manager.cc:1397] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:19:35.785881 14747 raft_consensus.cc:348] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } }
I20260925 12:19:35.785950 14747 raft_consensus.cc:374] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:19:35.785976 14747 raft_consensus.cc:729] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d3e88caaa34418876c3759d893f8fe, State: Initialized, Role: FOLLOWER
I20260925 12:19:35.786056 14747 consensus_queue.cc:260] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } }
I20260925 12:19:35.786162 14747 ts_tablet_manager.cc:1428] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:35.787053 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.782730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40589 (local address 127.0.0.1:58770)
0925 12:19:35.782780 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:35.782790 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:35.782823 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:35.783117 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:35.783118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:35.783120 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:19:35.783153 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.783155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.784281 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.784283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.785339 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.785342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.786886 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.786887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.786942 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.786945 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:19:35.786985 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 12:19:35.787012 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 12:19:35.787103 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.782796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58770 (local address 127.0.98.3:40589)
0925 12:19:35.782951 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:35.782955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:35.782966 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:19:35.783001 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:35.783003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:35.783018 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:35.783104 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:19:35.783268 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.784188 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.785420 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.786785 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.786884 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.786931 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:35.786996 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 12:19:35.787027 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260925 12:19:35.787724 15059 tablet_service.cc:1467] Processing CreateTablet for tablet 6b84ac78aca74f1d92b89e438b1b8b13 (DEFAULT_TABLE table=test-table [id=0348e1bdec044ee48d2d0800fe4a62ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:19:35.787995 15059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b84ac78aca74f1d92b89e438b1b8b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:19:35.788375 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.782731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46823 (local address 127.0.0.1:42582)
0925 12:19:35.782941 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:35.782963 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:35.783012 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:35.785053 (+ 2041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:35.785053 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:35.785056 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:19:35.785091 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.785093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.786285 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.786286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.787357 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.787359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.788217 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.788218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.788262 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.788265 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:19:35.788307 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 12:19:35.788332 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":89,"thread_start_us":66,"threads_started":1}
I20260925 12:19:35.788429 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.782863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42582 (local address 127.0.98.2:46823)
0925 12:19:35.784913 (+ 2050us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:35.784917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:35.784928 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:19:35.784962 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:35.784964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:35.784978 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:35.785054 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:19:35.785154 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.786180 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.787437 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.788141 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.788213 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.788262 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:35.788319 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 12:19:35.788354 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2018,"thread_start_us":82,"threads_started":1}
I20260925 12:19:35.788957 14927 tablet_service.cc:1467] Processing CreateTablet for tablet 6b84ac78aca74f1d92b89e438b1b8b13 (DEFAULT_TABLE table=test-table [id=0348e1bdec044ee48d2d0800fe4a62ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:19:35.789232 14927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b84ac78aca74f1d92b89e438b1b8b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:19:35.790450 15135 tablet_bootstrap.cc:492] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: Bootstrap starting.
I20260925 12:19:35.791092 15135 tablet_bootstrap.cc:654] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: Neither blocks nor log segments found. Creating new log.
I20260925 12:19:35.791517 15135 log.cc:824] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:35.792037 15137 tablet_bootstrap.cc:492] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5: Bootstrap starting.
I20260925 12:19:35.792294 15135 tablet_bootstrap.cc:492] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: No bootstrap required, opened a new log
I20260925 12:19:35.792372 15135 ts_tablet_manager.cc:1397] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:19:35.792690 15137 tablet_bootstrap.cc:654] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5: Neither blocks nor log segments found. Creating new log.
I20260925 12:19:35.793219 15137 log.cc:824] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:35.793913 15137 tablet_bootstrap.cc:492] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5: No bootstrap required, opened a new log
I20260925 12:19:35.793980 15137 ts_tablet_manager.cc:1397] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:19:35.794369 15135 raft_consensus.cc:348] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } }
I20260925 12:19:35.794528 15135 raft_consensus.cc:374] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:19:35.794559 15135 raft_consensus.cc:729] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d5988fdfc1455bac63cdafe131e287, State: Initialized, Role: FOLLOWER
I20260925 12:19:35.794664 15135 consensus_queue.cc:260] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } }
I20260925 12:19:35.794909 15135 ts_tablet_manager.cc:1428] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 12:19:35.794927 15125 heartbeater.cc:503] Master 127.0.98.62:32939 was elected leader, sending a full tablet report...
I20260925 12:19:35.795706 15137 raft_consensus.cc:348] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } }
I20260925 12:19:35.795845 15137 raft_consensus.cc:374] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:19:35.795890 15137 raft_consensus.cc:729] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ccb22dea9b04e6fa378297edce10fd5, State: Initialized, Role: FOLLOWER
I20260925 12:19:35.795995 15137 consensus_queue.cc:260] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } }
I20260925 12:19:35.796239 15137 ts_tablet_manager.cc:1428] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 12:19:35.796284 14993 heartbeater.cc:503] Master 127.0.98.62:32939 was elected leader, sending a full tablet report...
W20260925 12:19:35.885452 14994 tablet.cc:2369] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:19:35.947554 15139 raft_consensus.cc:482] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:19:35.947682 15139 raft_consensus.cc:504] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } }
I20260925 12:19:35.948113 15139 leader_election.cc:288] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ccb22dea9b04e6fa378297edce10fd5 (127.0.98.2:46823), 18d3e88caaa34418876c3759d893f8fe (127.0.98.1:44365)
I20260925 12:19:35.950615 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.948384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35565 (local address 127.0.98.2:46823)
0925 12:19:35.948433 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:35.948435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:35.948443 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:19:35.948472 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:35.948473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:35.948483 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:35.948544 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:19:35.948696 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.949134 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.949952 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.950294 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.950369 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.950379 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:35.950522 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 12:19:35.950557 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 12:19:35.950615 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.948268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46823 (local address 127.0.98.3:35565)
0925 12:19:35.948333 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:35.948344 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:35.948382 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:35.948548 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:35.948549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:35.948550 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:19:35.948629 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.948631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.949210 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.949211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.949868 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.949870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.950369 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.950370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.950448 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.950451 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:19:35.950506 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 12:19:35.950545 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 12:19:35.950682 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.948354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45631 (local address 127.0.98.1:44365)
0925 12:19:35.948394 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:35.948395 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:35.948531 (+ 136us) server_negotiation.cc:407] Connection header received
0925 12:19:35.948571 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:35.948572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:35.948584 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:35.948654 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:19:35.948791 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.949204 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.950025 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.950357 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.950432 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.950441 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:35.950581 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 12:19:35.950614 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:19:35.950734 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.948287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44365 (local address 127.0.98.3:45631)
0925 12:19:35.948450 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:35.948459 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:35.948521 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:35.948655 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:35.948656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:35.948658 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:19:35.948722 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.948723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.949281 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.949281 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.949936 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:35.949938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.950432 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.950433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:35.950498 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.950501 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:19:35.950569 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 12:19:35.950606 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
I20260925 12:19:35.950940 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b84ac78aca74f1d92b89e438b1b8b13" candidate_uuid: "12d5988fdfc1455bac63cdafe131e287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ccb22dea9b04e6fa378297edce10fd5" is_pre_election: true
I20260925 12:19:35.950964 14809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b84ac78aca74f1d92b89e438b1b8b13" candidate_uuid: "12d5988fdfc1455bac63cdafe131e287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d3e88caaa34418876c3759d893f8fe" is_pre_election: true
I20260925 12:19:35.951089 14809 raft_consensus.cc:2454] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12d5988fdfc1455bac63cdafe131e287 in term 0.
I20260925 12:19:35.951093 14947 raft_consensus.cc:2454] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12d5988fdfc1455bac63cdafe131e287 in term 0.
I20260925 12:19:35.951293 15014 leader_election.cc:302] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d5988fdfc1455bac63cdafe131e287, 18d3e88caaa34418876c3759d893f8fe; no voters:
I20260925 12:19:35.951453 15139 raft_consensus.cc:2781] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:19:35.951519 15139 raft_consensus.cc:482] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:19:35.951557 15139 raft_consensus.cc:3037] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:19:35.952266 15139 raft_consensus.cc:504] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } }
I20260925 12:19:35.952395 15139 leader_election.cc:288] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [CANDIDATE]: Term 1 election: Requested vote from peers 2ccb22dea9b04e6fa378297edce10fd5 (127.0.98.2:46823), 18d3e88caaa34418876c3759d893f8fe (127.0.98.1:44365)
I20260925 12:19:35.952524 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b84ac78aca74f1d92b89e438b1b8b13" candidate_uuid: "12d5988fdfc1455bac63cdafe131e287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ccb22dea9b04e6fa378297edce10fd5"
I20260925 12:19:35.952521 14809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b84ac78aca74f1d92b89e438b1b8b13" candidate_uuid: "12d5988fdfc1455bac63cdafe131e287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d3e88caaa34418876c3759d893f8fe"
I20260925 12:19:35.952607 14947 raft_consensus.cc:3037] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:19:35.952608 14809 raft_consensus.cc:3037] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:19:35.953532 14947 raft_consensus.cc:2454] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12d5988fdfc1455bac63cdafe131e287 in term 1.
I20260925 12:19:35.953716 15015 leader_election.cc:302] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d5988fdfc1455bac63cdafe131e287, 2ccb22dea9b04e6fa378297edce10fd5; no voters:
I20260925 12:19:35.953807 15139 raft_consensus.cc:2781] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:19:35.953868 14809 raft_consensus.cc:2454] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12d5988fdfc1455bac63cdafe131e287 in term 1.
I20260925 12:19:35.953969 15139 raft_consensus.cc:686] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [term 1 LEADER]: Becoming Leader. State: Replica: 12d5988fdfc1455bac63cdafe131e287, State: Running, Role: LEADER
I20260925 12:19:35.954072 15139 consensus_queue.cc:237] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } }
I20260925 12:19:35.954849 14667 catalog_manager.cc:5495] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12d5988fdfc1455bac63cdafe131e287 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "12d5988fdfc1455bac63cdafe131e287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } health_report { overall_health: UNKNOWN } } }
I20260925 12:19:35.966006 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:35.964178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58778 (local address 127.0.98.3:40589)
0925 12:19:35.964236 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:35.964237 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:35.964244 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:19:35.964284 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:35.964285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:35.964297 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:35.964350 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:19:35.964432 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.964826 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.965438 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:35.965601 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:35.965645 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:35.965764 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:19:35.965849 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:19:35.965899 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:35.965952 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 12:19:35.965967 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 12:19:35.967408 14947 raft_consensus.cc:1261] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [term 1 FOLLOWER]: Refusing update from remote peer 12d5988fdfc1455bac63cdafe131e287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:19:35.967445 14809 raft_consensus.cc:1261] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe [term 1 FOLLOWER]: Refusing update from remote peer 12d5988fdfc1455bac63cdafe131e287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:19:35.967712 15142 consensus_queue.cc:1035] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:19:35.967819 15139 consensus_queue.cc:1035] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:19:35.970294 15145 mvcc.cc:205] Tried to move back new op lower bound from 7333227626360565760 to 7333227626308665344. Current Snapshot: MvccSnapshot[applied={T|T < 7333227626360565760}]
I20260925 12:19:35.970999 15146 mvcc.cc:205] Tried to move back new op lower bound from 7333227626360565760 to 7333227626308665344. Current Snapshot: MvccSnapshot[applied={T|T < 7333227626360565760 or (T in {7333227626360565760})}]
I20260925 12:19:35.971657 14666 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:19:35.971740 14666 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:19:35.972426 14666 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a: Sending DeleteTablet for 3 replicas of tablet 6b84ac78aca74f1d92b89e438b1b8b13
I20260925 12:19:35.972755 14789 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b84ac78aca74f1d92b89e438b1b8b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:19:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40566
I20260925 12:19:35.972748 15059 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b84ac78aca74f1d92b89e438b1b8b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:19:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58770
I20260925 12:19:35.972762 14927 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b84ac78aca74f1d92b89e438b1b8b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:19:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42582
I20260925 12:19:35.972842 14864 tablet_replica.cc:330] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe: stopping tablet replica
I20260925 12:19:35.972896 14864 raft_consensus.cc:2229] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:19:35.972945 14864 raft_consensus.cc:2258] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:19:35.972955 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 14705
W20260925 12:19:35.978653 14653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44365 (error 108)
I20260925 12:19:35.978863 15155 tablet_replica.cc:330] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5: stopping tablet replica
I20260925 12:19:35.978957 15155 raft_consensus.cc:2229] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:19:35.979044 15155 raft_consensus.cc:2258] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:19:35.979182 15154 tablet_replica.cc:330] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: stopping tablet replica
I20260925 12:19:35.979259 15154 raft_consensus.cc:2229] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [term 1 LEADER]: Raft consensus shutting down.
W20260925 12:19:35.979262 15132 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:19:35.978953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44365 (local address 127.0.0.1:40580)
0925 12:19:35.978992 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:35.979000 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:35.979031 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:35.979217 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 12:19:35.979326 15154 raft_consensus.cc:2258] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 12:19:35.979360 14653 catalog_manager.cc:4584] TS 18d3e88caaa34418876c3759d893f8fe (127.0.98.1:44365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b84ac78aca74f1d92b89e438b1b8b13: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 12:19:35.979696 15155 ts_tablet_manager.cc:1905] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:19:35.979981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 14860
I20260925 12:19:35.983968 15154 ts_tablet_manager.cc:1905] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:19:35.985333 15154 ts_tablet_manager.cc:1918] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:19:35.985416 15154 log.cc:1196] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/6b84ac78aca74f1d92b89e438b1b8b13
I20260925 12:19:35.985714 15154 ts_tablet_manager.cc:1939] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: Deleting consensus metadata
W20260925 12:19:35.985803 15132 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:19:35.985667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46823 (local address 127.0.0.1:42590)
0925 12:19:35.985726 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:35.985771 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46823: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 12:19:35.986004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 14996
I20260925 12:19:35.986075 14651 catalog_manager.cc:4841] TS 12d5988fdfc1455bac63cdafe131e287 (127.0.98.3:40589): tablet 6b84ac78aca74f1d92b89e438b1b8b13 (table test-table [id=0348e1bdec044ee48d2d0800fe4a62ef]) successfully deleted
I20260925 12:19:35.990815 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 14634
I20260925 12:19:35.995810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32939
--webserver_interface=127.0.98.62
--webserver_port=34387
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:19:36.069034 15157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:36.069234 15157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:36.069270 15157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:36.070290 15157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:19:36.070346 15157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:36.070369 15157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:36.070387 15157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:19:36.070415 15157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:19:36.071785 15157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:36.072052 15157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:36.072263 15157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:36.076581 15163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:36.076555 15165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:36.076555 15162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:36.076722 15157 server_base.cc:1034] running on GCE node
I20260925 12:19:36.077023 15157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:36.077308 15157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:36.078459 15157 hybrid_clock.cc:648] HybridClock initialized: now 1790338776078447 us; error 28 us; skew 500 ppm
Sep 25 12:19:36 dist-test-slave-clmx krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:36.080639 15157 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:19:36.081013 15157 webserver.cc:460] Webserver started at http://127.0.98.62:34387/ using document root <none> and password file <none>
I20260925 12:19:36.081225 15157 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:36.081265 15157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:36.082433 15157 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:19:36.083042 15172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:36.083235 15157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:19:36.083302 15157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "43c4b6e343b64336b021ab9c29c13f3a"
format_stamp: "Formatted at 2026-09-25 12:19:34 on dist-test-slave-clmx"
server_key: "c1178a865042d05fb76af66a379c8484"
server_key_iv: "eb10a6dd3d57b448e4e3501071f1335c"
server_key_version: "encryptionkey@0"
I20260925 12:19:36.083647 15157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:36.105808 15157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:36.107713 15157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:36.107848 15157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:36.111573 15157 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32939
I20260925 12:19:36.111660 15224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32939 every 8 connection(s)
I20260925 12:19:36.111995 15157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:19:36.113412 15225 sys_catalog.cc:263] Verifying existing consensus state
I20260925 12:19:36.113960 15225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a: Bootstrap starting.
I20260925 12:19:36.120234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 15157
I20260925 12:19:36.120477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44365
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:32939
--webserver_port=32897
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:19:36.121765 15225 log.cc:824] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:36.123879 15225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:19:36.124150 15225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a: Bootstrap complete.
I20260925 12:19:36.126225 15225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4b6e343b64336b021ab9c29c13f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } }
I20260925 12:19:36.126446 15225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c4b6e343b64336b021ab9c29c13f3a, State: Initialized, Role: FOLLOWER
I20260925 12:19:36.126639 15225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4b6e343b64336b021ab9c29c13f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } }
I20260925 12:19:36.126732 15225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:19:36.126782 15225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:19:36.126829 15225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 2 FOLLOWER]: Advancing to term 3
I20260925 12:19:36.127738 15225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4b6e343b64336b021ab9c29c13f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } }
I20260925 12:19:36.127842 15225 leader_election.cc:302] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43c4b6e343b64336b021ab9c29c13f3a; no voters:
I20260925 12:19:36.127976 15225 leader_election.cc:288] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 12:19:36.128005 15229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 3 FOLLOWER]: Leader election won for term 3
I20260925 12:19:36.128113 15229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [term 3 LEADER]: Becoming Leader. State: Replica: 43c4b6e343b64336b021ab9c29c13f3a, State: Running, Role: LEADER
I20260925 12:19:36.128198 15225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:19:36.128186 15229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4b6e343b64336b021ab9c29c13f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } }
I20260925 12:19:36.128446 15230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "43c4b6e343b64336b021ab9c29c13f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4b6e343b64336b021ab9c29c13f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } } }
I20260925 12:19:36.128535 15230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [sys.catalog]: This master's current role is: LEADER
I20260925 12:19:36.129070 15231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43c4b6e343b64336b021ab9c29c13f3a. Latest consensus state: current_term: 3 leader_uuid: "43c4b6e343b64336b021ab9c29c13f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4b6e343b64336b021ab9c29c13f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } } }
I20260925 12:19:36.129149 15231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c4b6e343b64336b021ab9c29c13f3a [sys.catalog]: This master's current role is: LEADER
I20260925 12:19:36.129249 15234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:19:36.130072 15234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:19:36.130158 15234 catalog_manager.cc:1259] Loaded cluster ID: 6ea7f03a141b4dd9a9d6a3518892a723
I20260925 12:19:36.130185 15234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 12:19:36.130363 15234 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2f73f9a9ad google::LogMessage::Fail() at ??:0
@ 0x7f2f73f9d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f2f73f9a4de google::LogMessage::Flush() at ??:0
@ 0x7f2f73f9ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2f7bebb0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f2f7bec48b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2f75e680d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2f75e5d954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2f76cc16db start_thread at ??:0
@ 0x7f2f72a9861f clone at ??:0
W20260925 12:19:36.213447 15227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:36.213622 15227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:36.213645 15227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:36.214775 15227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:36.214811 15227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:36.214835 15227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:19:36.216284 15227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=32897
--tserver_master_addrs=127.0.98.62:32939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:36.216518 15227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:36.216761 15227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:36.221298 15264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:36.221459 15261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:36.221515 15227 server_base.cc:1034] running on GCE node
W20260925 12:19:36.221311 15262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:36.221787 15227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:36.222013 15227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:36.223155 15227 hybrid_clock.cc:648] HybridClock initialized: now 1790338776223135 us; error 36 us; skew 500 ppm
Sep 25 12:19:36 dist-test-slave-clmx krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:36.225680 15227 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:19:36.226068 15227 webserver.cc:460] Webserver started at http://127.0.98.1:32897/ using document root <none> and password file <none>
I20260925 12:19:36.226315 15227 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:36.226382 15227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:36.227660 15227 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:19:36.228338 15271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:36.228523 15227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:19:36.228610 15227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "18d3e88caaa34418876c3759d893f8fe"
format_stamp: "Formatted at 2026-09-25 12:19:34 on dist-test-slave-clmx"
server_key: "a1272ee60da76e59fdec3a0e9838de01"
server_key_iv: "ccf7711c75097f1f0057c4e4d3a249f1"
server_key_version: "encryptionkey@0"
I20260925 12:19:36.228914 15227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:36.244163 15227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:36.246049 15227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:36.246186 15227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:36.246372 15227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:19:36.246806 15278 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 12:19:36.248564 15227 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 12:19:36.248617 15227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 12:19:36.248657 15227 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 12:19:36.249456 15227 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 12:19:36.249498 15227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 12:19:36.249601 15278 tablet_bootstrap.cc:492] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe: Bootstrap starting.
I20260925 12:19:36.256903 15227 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44365
I20260925 12:19:36.256970 15385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44365 every 8 connection(s)
I20260925 12:19:36.257372 15227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:19:36.260337 15278 log.cc:824] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:36.262223 15278 tablet_bootstrap.cc:492] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:19:36.262488 15278 tablet_bootstrap.cc:492] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe: Bootstrap complete.
I20260925 12:19:36.263396 15278 ts_tablet_manager.cc:1397] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.006s
I20260925 12:19:36.264389 15278 raft_consensus.cc:348] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } }
I20260925 12:19:36.264599 15278 raft_consensus.cc:729] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d3e88caaa34418876c3759d893f8fe, State: Initialized, Role: FOLLOWER
I20260925 12:19:36.264730 15278 consensus_queue.cc:260] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } }
I20260925 12:19:36.264962 15278 ts_tablet_manager.cc:1428] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260925 12:19:36.265300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 15227
I20260925 12:19:36.265578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46823
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:32939
--webserver_port=44943
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:19:36.315889 15388 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:19:36.258046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:53637)
0925 12:19:36.258254 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:36.258300 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:36.258848 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:36.315713 (+ 56865us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":111,"threads_started":1}
W20260925 12:19:36.316195 15386 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32939 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:32939: Client connection negotiation failed: client connection to 127.0.98.62:32939: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 12:19:36.316527 15388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:36.316414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:51111)
0925 12:19:36.316465 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:36.316500 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 12:19:36.316870 15388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:36.316791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:56759)
0925 12:19:36.316836 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:36.316857 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 12:19:36.316991 15386 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 12:19:36.344473 15391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:36.344655 15391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:36.344688 15391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:36.345780 15391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:36.345839 15391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:36.345883 15391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:19:36.347227 15391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44943
--tserver_master_addrs=127.0.98.62:32939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:36.347447 15391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:36.347692 15391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:36.351886 15396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:36.351895 15399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:36.351895 15397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:36.352237 15391 server_base.cc:1034] running on GCE node
I20260925 12:19:36.352372 15391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:36.352594 15391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:36.353729 15391 hybrid_clock.cc:648] HybridClock initialized: now 1790338776353720 us; error 29 us; skew 500 ppm
Sep 25 12:19:36 dist-test-slave-clmx krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:36.355866 15391 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:19:36.356258 15391 webserver.cc:460] Webserver started at http://127.0.98.2:44943/ using document root <none> and password file <none>
I20260925 12:19:36.356468 15391 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:36.356509 15391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:36.357654 15391 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:19:36.358263 15406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:36.358428 15391 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:19:36.358517 15391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "2ccb22dea9b04e6fa378297edce10fd5"
format_stamp: "Formatted at 2026-09-25 12:19:34 on dist-test-slave-clmx"
server_key: "3e5a60367d816bf0403ca1929dbd09cd"
server_key_iv: "0198bbca33d89deeb0c43acf55e6ae1c"
server_key_version: "encryptionkey@0"
I20260925 12:19:36.358796 15391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:36.374943 15391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:36.376812 15391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:36.376968 15391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:36.377179 15391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:19:36.377552 15413 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 12:19:36.378798 15391 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 12:19:36.378850 15391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260925 12:19:36.378916 15391 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 12:19:36.379456 15391 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 12:19:36.379505 15391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 12:19:36.379560 15413 tablet_bootstrap.cc:492] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5: Bootstrap starting.
I20260925 12:19:36.381023 15413 log.cc:824] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:36.383045 15413 tablet_bootstrap.cc:492] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:19:36.383342 15413 tablet_bootstrap.cc:492] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5: Bootstrap complete.
I20260925 12:19:36.383514 15413 ts_tablet_manager.cc:1397] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260925 12:19:36.384791 15413 raft_consensus.cc:348] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } }
I20260925 12:19:36.385059 15413 raft_consensus.cc:729] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ccb22dea9b04e6fa378297edce10fd5, State: Initialized, Role: FOLLOWER
I20260925 12:19:36.385241 15413 consensus_queue.cc:260] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } }
I20260925 12:19:36.385514 15413 ts_tablet_manager.cc:1428] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20260925 12:19:36.386202 15391 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46823
I20260925 12:19:36.386247 15522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46823 every 8 connection(s)
I20260925 12:19:36.386503 15391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:19:36.387523 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:36.387031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:57719)
0925 12:19:36.387161 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:36.387216 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
W20260925 12:19:36.387789 15523 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32939 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:32939: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
I20260925 12:19:36.388060 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:36.387969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:35881)
0925 12:19:36.388017 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:36.388044 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 12:19:36.388356 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:36.388281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:36783)
0925 12:19:36.388322 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:36.388343 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 12:19:36.388494 15523 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 12:19:36.390233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 15391
I20260925 12:19:36.390466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40589
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:32939
--webserver_port=34625
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:19:36.467489 15526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:36.467717 15526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:36.467756 15526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:36.468880 15526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:36.468945 15526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:36.468990 15526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:19:36.470407 15526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34625
--tserver_master_addrs=127.0.98.62:32939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:36.470652 15526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:36.470875 15526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:19:36.475598 15526 server_base.cc:1034] running on GCE node
W20260925 12:19:36.475589 15531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:36.475574 15532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:36.475579 15534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:36.476042 15526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:36.476330 15526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:36.477481 15526 hybrid_clock.cc:648] HybridClock initialized: now 1790338776477471 us; error 37 us; skew 500 ppm
Sep 25 12:19:36 dist-test-slave-clmx krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:36.479784 15526 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:19:36.480166 15526 webserver.cc:460] Webserver started at http://127.0.98.3:34625/ using document root <none> and password file <none>
I20260925 12:19:36.480374 15526 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:36.480438 15526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:36.481675 15526 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:19:36.482235 15541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:36.482410 15526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:36.482503 15526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "12d5988fdfc1455bac63cdafe131e287"
format_stamp: "Formatted at 2026-09-25 12:19:35 on dist-test-slave-clmx"
server_key: "7e59f36c9ae20efc03b1b72a40a5cad9"
server_key_iv: "5dbcd933481d77c2feab6fafee078d6a"
server_key_version: "encryptionkey@0"
I20260925 12:19:36.482815 15526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:36.504261 15526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:36.506503 15526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:36.506660 15526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:36.506872 15526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:19:36.507149 15526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:19:36.507200 15526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:36.507237 15526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:19:36.507264 15526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:36.512809 15526 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40589
I20260925 12:19:36.512856 15654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40589 every 8 connection(s)
I20260925 12:19:36.513159 15526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:19:36.514086 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:36.513760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:53257)
0925 12:19:36.513893 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:36.513956 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
W20260925 12:19:36.514469 15655 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32939 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:32939: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
I20260925 12:19:36.514750 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:36.514665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:34743)
0925 12:19:36.514708 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:36.514733 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 12:19:36.515007 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:36.514940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:58485)
0925 12:19:36.514976 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:36.514995 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 12:19:36.515165 15655 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 12:19:36.515473 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 15526
I20260925 12:19:37.317504 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:37.317298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:52309)
0925 12:19:37.317408 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:37.317450 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260925 12:19:37.388908 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:37.388739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:45807)
0925 12:19:37.388849 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:37.388877 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":48,"threads_started":1}
I20260925 12:19:37.515892 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:37.515688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:44977)
0925 12:19:37.515801 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:37.515827 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
I20260925 12:19:37.603304 15662 raft_consensus.cc:482] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:19:37.603451 15662 raft_consensus.cc:504] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } }
I20260925 12:19:37.603849 15662 leader_election.cc:288] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12d5988fdfc1455bac63cdafe131e287 (127.0.98.3:40589), 18d3e88caaa34418876c3759d893f8fe (127.0.98.1:44365)
Sep 25 12:19:37 dist-test-slave-clmx krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 25 12:19:37 dist-test-slave-clmx krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:19:37.621038 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:37.604112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58913 (local address 127.0.98.1:44365)
0925 12:19:37.604236 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:37.604241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:37.604254 (+ 13us) server_negotiation.cc:407] Connection header received
0925 12:19:37.604347 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:37.604350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:37.604372 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:37.604463 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:37.605467 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:37.606765 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:37.608254 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:37.608874 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:37.608986 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:37.617669 (+ 8683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:37.617672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:37.617672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:37.617681 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:37.619449 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:37.619795 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:37.619797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:37.619797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:37.619877 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:37.620166 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:37.620166 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:37.620166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:37.620452 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:37.620530 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:37.620839 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 12:19:37.620950 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":49,"threads_started":1}
I20260925 12:19:37.621047 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:37.604076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42811 (local address 127.0.98.3:40589)
0925 12:19:37.604275 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:37.604284 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:37.604314 (+ 30us) server_negotiation.cc:407] Connection header received
0925 12:19:37.604815 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:37.604817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:37.604855 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:37.604952 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:37.605465 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:37.606765 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:37.608254 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:37.608874 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:37.608986 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:37.617319 (+ 8333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:37.617323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:37.617323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:37.617339 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:37.619230 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:37.619742 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:37.619744 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:37.619744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:37.619876 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:37.620166 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:37.620166 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:37.620167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:37.620451 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:37.620530 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:37.620831 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 12:19:37.620949 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":56,"threads_started":1}
I20260925 12:19:37.621061 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:37.603993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44365 (local address 127.0.98.2:58913)
0925 12:19:37.604110 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:37.604120 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:37.604204 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:37.604506 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:37.604509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:37.604777 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:37.605344 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:37.605352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:37.606867 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:37.606867 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:37.608168 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:37.608171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:37.608969 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:37.608970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:37.609108 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:37.615863 (+ 6755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:37.615880 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:37.617557 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:37.619512 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:37.619513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:37.619514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:37.619733 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:37.619943 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:37.619943 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:37.619943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:37.620083 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:37.620520 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:37.620525 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:37.620662 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 12:19:37.620758 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 12:19:37.620946 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":88,"thread_start_us":55,"threads_started":1}
I20260925 12:19:37.621086 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:37.603955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40589 (local address 127.0.98.2:42811)
0925 12:19:37.604004 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:37.604028 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:37.604204 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:37.604969 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:37.604970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:37.605150 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:37.605344 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:37.605352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:37.606859 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:37.606860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:37.608168 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:37.608171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:37.608970 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:37.608971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:37.609108 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:37.615895 (+ 6787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:37.615900 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:37.617216 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:37.619346 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:37.619351 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:37.619353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:37.619665 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:37.619943 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:37.619943 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:37.619943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:37.620083 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:37.620520 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:37.620525 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:37.620662 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 12:19:37.620758 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 12:19:37.620993 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":189}
W20260925 12:19:37.621814 15407 leader_election.cc:341] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12d5988fdfc1455bac63cdafe131e287 (127.0.98.3:40589): Not found: Tablet not found: 6b84ac78aca74f1d92b89e438b1b8b13
I20260925 12:19:37.622107 15340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b84ac78aca74f1d92b89e438b1b8b13" candidate_uuid: "2ccb22dea9b04e6fa378297edce10fd5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "18d3e88caaa34418876c3759d893f8fe" is_pre_election: true
I20260925 12:19:37.622237 15340 raft_consensus.cc:2454] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ccb22dea9b04e6fa378297edce10fd5 in term 1.
I20260925 12:19:37.622493 15409 leader_election.cc:302] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18d3e88caaa34418876c3759d893f8fe, 2ccb22dea9b04e6fa378297edce10fd5; no voters: 12d5988fdfc1455bac63cdafe131e287
I20260925 12:19:37.622607 15662 raft_consensus.cc:2781] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 12:19:37.622679 15662 raft_consensus.cc:482] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:19:37.622707 15662 raft_consensus.cc:3037] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 12:19:37.623980 15662 raft_consensus.cc:504] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } }
I20260925 12:19:37.624131 15662 leader_election.cc:288] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [CANDIDATE]: Term 2 election: Requested vote from peers 12d5988fdfc1455bac63cdafe131e287 (127.0.98.3:40589), 18d3e88caaa34418876c3759d893f8fe (127.0.98.1:44365)
I20260925 12:19:37.624284 15340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b84ac78aca74f1d92b89e438b1b8b13" candidate_uuid: "2ccb22dea9b04e6fa378297edce10fd5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "18d3e88caaa34418876c3759d893f8fe"
I20260925 12:19:37.624361 15340 raft_consensus.cc:3037] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe [term 1 FOLLOWER]: Advancing to term 2
W20260925 12:19:37.624405 15407 leader_election.cc:341] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 12d5988fdfc1455bac63cdafe131e287 (127.0.98.3:40589): Not found: Tablet not found: 6b84ac78aca74f1d92b89e438b1b8b13
I20260925 12:19:37.625335 15340 raft_consensus.cc:2454] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ccb22dea9b04e6fa378297edce10fd5 in term 2.
I20260925 12:19:37.625512 15409 leader_election.cc:302] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18d3e88caaa34418876c3759d893f8fe, 2ccb22dea9b04e6fa378297edce10fd5; no voters: 12d5988fdfc1455bac63cdafe131e287
I20260925 12:19:37.625626 15662 raft_consensus.cc:2781] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 12:19:37.625747 15662 raft_consensus.cc:686] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [term 2 LEADER]: Becoming Leader. State: Replica: 2ccb22dea9b04e6fa378297edce10fd5, State: Running, Role: LEADER
I20260925 12:19:37.625831 15662 consensus_queue.cc:237] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } }
I20260925 12:19:37.626012 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:37.625929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:47523)
0925 12:19:37.625966 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:37.625987 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 12:19:38.097021 15407 consensus_peers.cc:487] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 -> Peer 12d5988fdfc1455bac63cdafe131e287 (127.0.98.3:40589): Couldn't send request to peer 12d5988fdfc1455bac63cdafe131e287. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b84ac78aca74f1d92b89e438b1b8b13. This is attempt 1: this message will repeat every 5th retry.
I20260925 12:19:38.116861 15340 raft_consensus.cc:1261] T 6b84ac78aca74f1d92b89e438b1b8b13 P 18d3e88caaa34418876c3759d893f8fe [term 2 FOLLOWER]: Refusing update from remote peer 2ccb22dea9b04e6fa378297edce10fd5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260925 12:19:38.117221 15662 consensus_queue.cc:1035] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 12:19:38.118129 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:38.117955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:45499)
0925 12:19:38.118034 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:38.118086 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67,"thread_start_us":51,"threads_started":1}
I20260925 12:19:38.118992 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:38.118868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:50307)
0925 12:19:38.118904 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:38.118957 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 12:19:38.119087 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:38.118980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:38999)
0925 12:19:38.119023 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:38.119050 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 12:19:38.516446 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:38.516247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:60287)
0925 12:19:38.516373 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:38.516407 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260925 12:19:38.680191 15675 ts_tablet_manager.cc:927] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: Initiating tablet copy from peer 2ccb22dea9b04e6fa378297edce10fd5 (127.0.98.2:46823)
I20260925 12:19:38.680390 15675 tablet_copy_client.cc:316] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.2:46823
Sep 25 12:19:38 dist-test-slave-clmx krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260925 12:19:38.688223 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:38.680741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56579 (local address 127.0.98.2:46823)
0925 12:19:38.680833 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:38.680837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:38.680845 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:19:38.680866 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:38.680867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:38.680878 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:38.680937 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:38.681351 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:38.681983 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:38.682748 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:38.683121 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:38.683178 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:38.685247 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:38.685254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:38.685254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:38.685260 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:38.686829 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:38.687249 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:38.687252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:38.687252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:38.687326 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:38.687556 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:38.687557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:38.687557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:38.687769 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:38.687861 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:38.688043 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 12:19:38.688115 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":37,"threads_started":1}
I20260925 12:19:38.688226 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:38.680665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46823 (local address 127.0.98.3:56579)
0925 12:19:38.680715 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:38.680733 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:38.680803 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:38.680939 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:38.680941 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:38.681215 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:38.681282 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:38.681285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:38.682053 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:38.682054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:38.682679 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:38.682681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:38.683192 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:38.683193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:38.683264 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:38.683776 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:38.683780 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:38.685164 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:38.686932 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:38.686938 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:38.686941 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:38.687143 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:38.687402 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:38.687403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:38.687403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:38.687482 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:38.687840 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:38.687847 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:38.687898 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 12:19:38.688000 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 12:19:38.688114 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 12:19:38.688663 15496 tablet_copy_service.cc:140] P 2ccb22dea9b04e6fa378297edce10fd5: Received BeginTabletCopySession request for tablet 6b84ac78aca74f1d92b89e438b1b8b13 from peer 12d5988fdfc1455bac63cdafe131e287 ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56579)
I20260925 12:19:38.688721 15496 tablet_copy_service.cc:161] P 2ccb22dea9b04e6fa378297edce10fd5: Beginning new tablet copy session on tablet 6b84ac78aca74f1d92b89e438b1b8b13 from peer 12d5988fdfc1455bac63cdafe131e287 at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56579: session id = 12d5988fdfc1455bac63cdafe131e287-6b84ac78aca74f1d92b89e438b1b8b13
I20260925 12:19:38.689137 15496 tablet_copy_source_session.cc:205] T 6b84ac78aca74f1d92b89e438b1b8b13 P 2ccb22dea9b04e6fa378297edce10fd5: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 12:19:38.689955 15675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b84ac78aca74f1d92b89e438b1b8b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:19:38.692394 15675 tablet_copy_client.cc:791] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: tablet copy: Starting download of 0 data blocks...
I20260925 12:19:38.692584 15675 tablet_copy_client.cc:655] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: tablet copy: Starting download of 1 WAL segments...
I20260925 12:19:38.693315 15675 tablet_copy_client.cc:531] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 12:19:38.693935 15675 tablet_bootstrap.cc:492] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: Bootstrap starting.
I20260925 12:19:38.695128 15675 log.cc:824] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:38.697243 15675 tablet_bootstrap.cc:492] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:19:38.697407 15675 tablet_bootstrap.cc:492] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: Bootstrap complete.
I20260925 12:19:38.697481 15675 ts_tablet_manager.cc:1397] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 12:19:38.698686 15675 raft_consensus.cc:348] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } }
I20260925 12:19:38.698814 15675 raft_consensus.cc:729] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d5988fdfc1455bac63cdafe131e287, State: Initialized, Role: FOLLOWER
I20260925 12:19:38.698913 15675 consensus_queue.cc:260] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d5988fdfc1455bac63cdafe131e287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40589 } } peers { permanent_uuid: "2ccb22dea9b04e6fa378297edce10fd5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46823 } } peers { permanent_uuid: "18d3e88caaa34418876c3759d893f8fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44365 } }
I20260925 12:19:38.699174 15675 ts_tablet_manager.cc:1428] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 12:19:38.699407 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:38.699331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:36417)
0925 12:19:38.699359 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:38.699381 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260925 12:19:38.699415 15496 tablet_copy_service.cc:342] P 2ccb22dea9b04e6fa378297edce10fd5: Request end of tablet copy session 12d5988fdfc1455bac63cdafe131e287-6b84ac78aca74f1d92b89e438b1b8b13 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56579
I20260925 12:19:38.699455 15496 tablet_copy_service.cc:433] P 2ccb22dea9b04e6fa378297edce10fd5: ending tablet copy session 12d5988fdfc1455bac63cdafe131e287-6b84ac78aca74f1d92b89e438b1b8b13 on tablet 6b84ac78aca74f1d92b89e438b1b8b13 with peer 12d5988fdfc1455bac63cdafe131e287
I20260925 12:19:38.699743 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:38.699649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:38975)
0925 12:19:38.699698 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:38.699723 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 12:19:39.119693 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:39.119458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:38801)
0925 12:19:39.119558 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:39.119588 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":65,"threads_started":1}
I20260925 12:19:39.119719 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:39.119480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:45589)
0925 12:19:39.119580 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:39.119653 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":61,"threads_started":1}
I20260925 12:19:39.188781 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:39.188650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:33413)
0925 12:19:39.188702 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:39.188732 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 12:19:39.188781 15609 raft_consensus.cc:1203] T 6b84ac78aca74f1d92b89e438b1b8b13 P 12d5988fdfc1455bac63cdafe131e287 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260925 12:19:39.189189 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:39.189085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:35853)
0925 12:19:39.189132 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:39.189156 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 12:19:40.120257 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:40.120065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:34285)
0925 12:19:40.120178 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:40.120211 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260925 12:19:40.189332 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:40.189134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:39707)
0925 12:19:40.189257 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:40.189287 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260925 12:19:40.189779 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:40.189606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:44693)
0925 12:19:40.189707 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:40.189742 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":69,"threads_started":1}
I20260925 12:19:41.120908 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:41.120715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:55589)
0925 12:19:41.120831 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:41.120864 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260925 12:19:41.189906 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:41.189701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:57059)
0925 12:19:41.189819 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:41.189851 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":54,"threads_started":1}
I20260925 12:19:41.190315 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:41.190163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:35387)
0925 12:19:41.190253 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:41.190281 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":45,"threads_started":1}
I20260925 12:19:42.121657 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:42.121447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:49433)
0925 12:19:42.121577 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:42.121611 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":57,"threads_started":1}
I20260925 12:19:42.190449 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:42.190280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:46433)
0925 12:19:42.190379 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:42.190407 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":49,"threads_started":1}
I20260925 12:19:42.190912 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:42.190753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:58021)
0925 12:19:42.190848 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:42.190877 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":45,"threads_started":1}
I20260925 12:19:43.122447 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:43.122225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:48301)
0925 12:19:43.122369 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:43.122403 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":72,"threads_started":1}
I20260925 12:19:43.191110 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:43.190917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:52271)
0925 12:19:43.191033 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:43.191064 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260925 12:19:43.191502 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:43.191299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:54789)
0925 12:19:43.191425 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:43.191460 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260925 12:19:44.123199 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:44.122953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:34109)
0925 12:19:44.123110 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:44.123143 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
I20260925 12:19:44.192034 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:44.191848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:45225)
0925 12:19:44.191962 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:44.191991 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260925 12:19:44.192066 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:44.191902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:45217)
0925 12:19:44.192004 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:44.192029 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":44,"threads_started":1}
I20260925 12:19:45.123903 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:45.123698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:50779)
0925 12:19:45.123824 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:45.123859 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":52,"threads_started":1}
I20260925 12:19:45.192627 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:45.192414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:57207)
0925 12:19:45.192543 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:45.192583 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":55,"threads_started":1}
I20260925 12:19:45.192652 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:45.192439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:52601)
0925 12:19:45.192581 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:45.192614 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260925 12:19:46.124604 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:46.124392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:56079)
0925 12:19:46.124523 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:46.124547 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260925 12:19:46.193272 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:46.193048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:48837)
0925 12:19:46.193206 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:46.193234 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":64,"threads_started":1}
I20260925 12:19:46.193269 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:46.193044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:33207)
0925 12:19:46.193184 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:46.193221 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260925 12:19:47.125379 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:47.125036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:51595)
0925 12:19:47.125305 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:47.125338 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":58,"threads_started":1}
I20260925 12:19:47.193888 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:47.193685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:35879)
0925 12:19:47.193805 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:47.193846 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":54,"threads_started":1}
I20260925 12:19:47.193913 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:47.193716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:40999)
0925 12:19:47.193843 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:47.193873 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260925 12:19:48.125993 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:48.125804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:48887)
0925 12:19:48.125915 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:48.125949 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260925 12:19:48.194518 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:48.194318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:57571)
0925 12:19:48.194437 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:48.194478 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":50,"threads_started":1}
I20260925 12:19:48.194535 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:48.194327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:40505)
0925 12:19:48.194466 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:48.194497 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260925 12:19:49.126572 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:49.126380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:55633)
0925 12:19:49.126495 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:49.126527 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260925 12:19:49.195147 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:49.194961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:33447)
0925 12:19:49.195079 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:49.195104 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260925 12:19:49.195150 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:49.194967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:54179)
0925 12:19:49.195085 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:49.195111 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
I20260925 12:19:50.127173 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:50.126984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:57953)
0925 12:19:50.127101 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:50.127131 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":53,"threads_started":1}
I20260925 12:19:50.195868 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:50.195682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:53355)
0925 12:19:50.195798 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:50.195825 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
I20260925 12:19:50.195991 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:50.195796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:46817)
0925 12:19:50.195909 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:50.195943 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":73,"threads_started":1}
I20260925 12:19:51.127987 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:51.127786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:42715)
0925 12:19:51.127912 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:51.127942 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260925 12:19:51.196451 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:51.196231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:56531)
0925 12:19:51.196378 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:51.196405 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260925 12:19:51.196614 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:51.196398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:60881)
0925 12:19:51.196501 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:51.196565 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":64,"threads_started":1}
I20260925 12:19:51.523162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 15227
I20260925 12:19:51.528055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 15391
I20260925 12:19:51.532313 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 15526
2026-09-25T12:19:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18131 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:19:51 dist-test-slave-clmx krb5kdc[15722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:19:51 dist-test-slave-clmx krb5kdc[15722](info): set up 2 sockets
Sep 25 12:19:51 dist-test-slave-clmx krb5kdc[15722](info): commencing operation
krb5kdc: starting...
W20260925 12:19:52.551409 392 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:19:52 dist-test-slave-clmx krb5kdc[15722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:19:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:19:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:19:52.657721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38775
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:19:52.733192 15735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:52.733379 15735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:52.733412 15735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:52.734597 15735 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 12:19:52.734654 15735 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 12:19:52.734674 15735 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 12:19:52.734694 15735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:19:52.734723 15735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:52.734750 15735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:52.734779 15735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:19:52.734809 15735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:19:52.736219 15735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:52.736464 15735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:52.736678 15735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:52.741413 15740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:52.741446 15741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:52.741490 15735 server_base.cc:1034] running on GCE node
W20260925 12:19:52.741449 15743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:52.741765 15735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:52.742009 15735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:52.743163 15735 hybrid_clock.cc:648] HybridClock initialized: now 1790338792743157 us; error 32 us; skew 500 ppm
Sep 25 12:19:52 dist-test-slave-clmx krb5kdc[15722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:52.745551 15735 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:19:52.746006 15735 webserver.cc:460] Webserver started at http://127.0.98.62:37577/ using document root <none> and password file <none>
I20260925 12:19:52.746241 15735 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:52.746307 15735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:52.746433 15735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:19:52.747360 15735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "484a907addb347f89c3c247bd8e796d9"
format_stamp: "Formatted at 2026-09-25 12:19:52 on dist-test-slave-clmx"
server_key: "c65e639ef7b280f6c67c672faf5895f5"
server_key_iv: "3cf47fccc2562dcf12e366aa786d88de"
server_key_version: "encryptionkey@0"
I20260925 12:19:52.747738 15735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "484a907addb347f89c3c247bd8e796d9"
format_stamp: "Formatted at 2026-09-25 12:19:52 on dist-test-slave-clmx"
server_key: "c65e639ef7b280f6c67c672faf5895f5"
server_key_iv: "3cf47fccc2562dcf12e366aa786d88de"
server_key_version: "encryptionkey@0"
I20260925 12:19:52.748988 15735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:19:52.749625 15750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:52.749822 15735 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:52.749912 15735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "484a907addb347f89c3c247bd8e796d9"
format_stamp: "Formatted at 2026-09-25 12:19:52 on dist-test-slave-clmx"
server_key: "c65e639ef7b280f6c67c672faf5895f5"
server_key_iv: "3cf47fccc2562dcf12e366aa786d88de"
server_key_version: "encryptionkey@0"
I20260925 12:19:52.749991 15735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:52.756675 15735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:52.758775 15735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:52.758921 15735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:52.762820 15735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38775
I20260925 12:19:52.762848 15802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38775 every 8 connection(s)
I20260925 12:19:52.763185 15735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:19:52.764005 15803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:19:52.765996 15803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9: Bootstrap starting.
I20260925 12:19:52.766533 15803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9: Neither blocks nor log segments found. Creating new log.
I20260925 12:19:52.766808 15803 log.cc:824] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:52.767402 15803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9: No bootstrap required, opened a new log
I20260925 12:19:52.768900 15803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484a907addb347f89c3c247bd8e796d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38775 } }
I20260925 12:19:52.769019 15803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:19:52.769062 15803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484a907addb347f89c3c247bd8e796d9, State: Initialized, Role: FOLLOWER
I20260925 12:19:52.769194 15803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484a907addb347f89c3c247bd8e796d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38775 } }
I20260925 12:19:52.769255 15803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:19:52.769294 15803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:19:52.769337 15803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:19:52.769786 15803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484a907addb347f89c3c247bd8e796d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38775 } }
I20260925 12:19:52.769892 15803 leader_election.cc:302] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 484a907addb347f89c3c247bd8e796d9; no voters:
I20260925 12:19:52.770057 15803 leader_election.cc:288] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:19:52.770095 15806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:19:52.770269 15806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 1 LEADER]: Becoming Leader. State: Replica: 484a907addb347f89c3c247bd8e796d9, State: Running, Role: LEADER
I20260925 12:19:52.770313 15803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:19:52.770347 15806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484a907addb347f89c3c247bd8e796d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38775 } }
I20260925 12:19:52.770973 15807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "484a907addb347f89c3c247bd8e796d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484a907addb347f89c3c247bd8e796d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38775 } } }
I20260925 12:19:52.770984 15808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 484a907addb347f89c3c247bd8e796d9. Latest consensus state: current_term: 1 leader_uuid: "484a907addb347f89c3c247bd8e796d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484a907addb347f89c3c247bd8e796d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38775 } } }
I20260925 12:19:52.771071 15807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [sys.catalog]: This master's current role is: LEADER
I20260925 12:19:52.771075 15808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [sys.catalog]: This master's current role is: LEADER
I20260925 12:19:52.771227 15815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:19:52.771736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 15735
I20260925 12:19:52.771775 15815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:19:52.771826 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:19:52.772033 392 external_mini_cluster.cc:1428] Setting key ec7449b4dd98aadcec564d058572bfdf
I20260925 12:19:52.773432 15815 catalog_manager.cc:1347] Generated new cluster ID: 31dbbae372d64aaa9bc4f6fa439eb23c
I20260925 12:19:52.773481 15815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:19:52 dist-test-slave-clmx krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:52.783197 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:52.772532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33262 (local address 127.0.98.62:38775)
0925 12:19:52.772687 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:52.772695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:52.772870 (+ 175us) server_negotiation.cc:407] Connection header received
0925 12:19:52.773085 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:52.773091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:52.773144 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:52.773282 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:52.773974 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:52.775412 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:52.776473 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:52.777069 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:52.777189 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:52.779792 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:52.779796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:52.779796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:52.779806 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:52.781605 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:52.782008 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:52.782011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:52.782011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:52.782111 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:52.782319 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:52.782320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:52.782320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:52.782579 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:52.782693 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:52.782854 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 12:19:52.782988 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20260925 12:19:52.788844 15815 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:19:52.789305 15815 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:19:52.810966 15815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9: Generated new TSK 0
I20260925 12:19:52.811131 15815 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:19:52.837513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38775
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:19:52.928361 15829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:52.928530 15829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:52.928551 15829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:52.929553 15829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:52.929581 15829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:52.929603 15829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:19:52.930892 15829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:52.931048 15829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:52.931210 15829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:52.935832 15834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:52.935840 15837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:52.935856 15835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:52.936018 15829 server_base.cc:1034] running on GCE node
I20260925 12:19:52.936296 15829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:52.936520 15829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:52.937654 15829 hybrid_clock.cc:648] HybridClock initialized: now 1790338792937643 us; error 27 us; skew 500 ppm
Sep 25 12:19:52 dist-test-slave-clmx krb5kdc[15722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:52.940260 15829 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:19:52.940645 15829 webserver.cc:460] Webserver started at http://127.0.98.1:40007/ using document root <none> and password file <none>
I20260925 12:19:52.940862 15829 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:52.940936 15829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:52.941059 15829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:19:52.942085 15829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "2c97a4eabb5d4282aae54e4aa821dd35"
format_stamp: "Formatted at 2026-09-25 12:19:52 on dist-test-slave-clmx"
server_key: "f178376905e9010a59782a68ff64e4b0"
server_key_iv: "60849bfa8ab56febadb62f85fa632a65"
server_key_version: "encryptionkey@0"
I20260925 12:19:52.942408 15829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2c97a4eabb5d4282aae54e4aa821dd35"
format_stamp: "Formatted at 2026-09-25 12:19:52 on dist-test-slave-clmx"
server_key: "f178376905e9010a59782a68ff64e4b0"
server_key_iv: "60849bfa8ab56febadb62f85fa632a65"
server_key_version: "encryptionkey@0"
I20260925 12:19:52.943715 15829 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 12:19:52.944387 15844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:52.944542 15829 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:19:52.944634 15829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "2c97a4eabb5d4282aae54e4aa821dd35"
format_stamp: "Formatted at 2026-09-25 12:19:52 on dist-test-slave-clmx"
server_key: "f178376905e9010a59782a68ff64e4b0"
server_key_iv: "60849bfa8ab56febadb62f85fa632a65"
server_key_version: "encryptionkey@0"
I20260925 12:19:52.944725 15829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:52.953410 15829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:52.955054 15829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:52.955196 15829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:52.955396 15829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:19:52.955734 15829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:19:52.955783 15829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:52.955817 15829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:19:52.955842 15829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:52.960878 15829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44443
I20260925 12:19:52.960950 15957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44443 every 8 connection(s)
I20260925 12:19:52.961208 15829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:19:52.962049 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 15829
I20260925 12:19:52.962139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:19:52.962317 392 external_mini_cluster.cc:1428] Setting key db521d432fc32b2073520042d54ece9a
Sep 25 12:19:52 dist-test-slave-clmx krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 12:19:52.970988 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:52.961828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47003 (local address 127.0.98.62:38775)
0925 12:19:52.961874 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:52.961876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:52.962043 (+ 167us) server_negotiation.cc:407] Connection header received
0925 12:19:52.962623 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:52.962624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:52.962636 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:52.962709 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:52.963667 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:52.964445 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:52.965737 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:52.966091 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:52.966166 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:52.968149 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:52.968151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:52.968151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:52.968160 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:52.969657 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:52.970032 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:52.970033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:52.970034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:52.970092 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:52.970361 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:52.970362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:52.970362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:52.970503 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:52.970567 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:52.970821 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 12:19:52.970907 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 12:19:52.971113 15960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:52.961797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.1:47003)
0925 12:19:52.961919 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:52.961963 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:52.962530 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:52.962769 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:52.962772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:52.963333 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:52.963539 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:52.963544 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:52.964541 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:52.964542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:52.965655 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:52.965657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:52.966172 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:52.966173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:52.966264 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:52.966764 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:52.966771 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:52.968059 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:52.969755 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:52.969761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:52.969762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:52.969955 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:52.970163 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:52.970163 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:52.970164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:52.970294 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:52.970562 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:52.970567 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:52.970659 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 12:19:52.970797 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 12:19:52.970928 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":57,"threads_started":1}
I20260925 12:19:52.971674 15958 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38775
I20260925 12:19:52.971774 15958 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:52.971907 15958 heartbeater.cc:511] Master 127.0.98.62:38775 requested a full tablet report, sending...
I20260925 12:19:52.972421 15767 ts_manager.cc:194] Registered new tserver with Master: 2c97a4eabb5d4282aae54e4aa821dd35 (127.0.98.1:44443)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 12:19:52.973001 15767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:19:52.986617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38775
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:19:53.062457 15963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:53.062625 15963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:53.062646 15963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:53.063709 15963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:53.063743 15963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:53.063768 15963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:19:53.065083 15963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:53.065243 15963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:53.065393 15963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:53.069837 15969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:53.069868 15971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:53.069869 15968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:53.069985 15963 server_base.cc:1034] running on GCE node
I20260925 12:19:53.070243 15963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:53.070469 15963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:53.071659 15963 hybrid_clock.cc:648] HybridClock initialized: now 1790338793071644 us; error 29 us; skew 500 ppm
Sep 25 12:19:53 dist-test-slave-clmx krb5kdc[15722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:53.073971 15963 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:19:53.074352 15963 webserver.cc:460] Webserver started at http://127.0.98.2:40879/ using document root <none> and password file <none>
I20260925 12:19:53.074571 15963 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:53.074613 15963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:53.074759 15963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:19:53.075734 15963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "1c0e6524debb4ecf9571f9b526597f17"
format_stamp: "Formatted at 2026-09-25 12:19:53 on dist-test-slave-clmx"
server_key: "f0d753a38533a4cdb53ae9c2e8d4effd"
server_key_iv: "a9409ea16e4d9841c82a0665e5f15efe"
server_key_version: "encryptionkey@0"
I20260925 12:19:53.076043 15963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1c0e6524debb4ecf9571f9b526597f17"
format_stamp: "Formatted at 2026-09-25 12:19:53 on dist-test-slave-clmx"
server_key: "f0d753a38533a4cdb53ae9c2e8d4effd"
server_key_iv: "a9409ea16e4d9841c82a0665e5f15efe"
server_key_version: "encryptionkey@0"
I20260925 12:19:53.077167 15963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:19:53.077821 15978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:53.077994 15963 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:19:53.078059 15963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "1c0e6524debb4ecf9571f9b526597f17"
format_stamp: "Formatted at 2026-09-25 12:19:53 on dist-test-slave-clmx"
server_key: "f0d753a38533a4cdb53ae9c2e8d4effd"
server_key_iv: "a9409ea16e4d9841c82a0665e5f15efe"
server_key_version: "encryptionkey@0"
I20260925 12:19:53.078146 15963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:53.106014 15963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:53.107733 15963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:53.107884 15963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:53.108115 15963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:19:53.108407 15963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:19:53.108461 15963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:53.108493 15963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:19:53.108520 15963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:53.113895 15963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39615
I20260925 12:19:53.113994 16091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39615 every 8 connection(s)
I20260925 12:19:53.114269 15963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:19:53 dist-test-slave-clmx krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:53.121202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 15963
I20260925 12:19:53.121275 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:19:53.121460 392 external_mini_cluster.cc:1428] Setting key dafd7989af198ee79f10c3e8c2fec5d7
I20260925 12:19:53.122915 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:53.114940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38289 (local address 127.0.98.62:38775)
0925 12:19:53.115044 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:53.115047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:53.115218 (+ 171us) server_negotiation.cc:407] Connection header received
0925 12:19:53.115753 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:53.115755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:53.115767 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:53.115818 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:53.116678 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.117151 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.118082 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.118302 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.118376 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:53.120318 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:53.120319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:53.120319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:53.120326 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:53.121580 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:53.122054 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:53.122055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:53.122055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:53.122116 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:53.122384 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:53.122385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:53.122385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:53.122473 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:53.122520 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:53.122801 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 12:19:53.122858 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 12:19:53.123164 16094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:53.114923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.2:38289)
0925 12:19:53.115110 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:53.115139 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:53.115598 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:53.115883 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:53.115887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:53.116389 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:53.116554 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:53.116558 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.117248 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.117249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:53.117994 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:53.117996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.118391 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.118392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:53.118500 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:53.118964 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:53.118971 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:53.120231 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:53.121667 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:53.121671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:53.121673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:53.121928 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:53.122180 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:53.122181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:53.122181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:53.122322 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:53.122531 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:53.122537 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:53.122682 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 12:19:53.122784 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 12:19:53.122933 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260925 12:19:53.123593 16092 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38775
I20260925 12:19:53.123699 16092 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:53.123875 16092 heartbeater.cc:511] Master 127.0.98.62:38775 requested a full tablet report, sending...
I20260925 12:19:53.124239 15767 ts_manager.cc:194] Registered new tserver with Master: 1c0e6524debb4ecf9571f9b526597f17 (127.0.98.2:39615)
I20260925 12:19:53.124604 15767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38289
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:19:53.143810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38775
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:19:53.220224 16097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:19:53.220398 16097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:19:53.220422 16097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:19:53.221441 16097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:19:53.221472 16097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:19:53.221496 16097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:19:53.222765 16097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:19:53.222919 16097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:19:53.223079 16097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:19:53.227677 16102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:53.227903 16097 server_base.cc:1034] running on GCE node
W20260925 12:19:53.227692 16103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:19:53.227698 16105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:19:53.228199 16097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:19:53.228447 16097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:19:53.229583 16097 hybrid_clock.cc:648] HybridClock initialized: now 1790338793229560 us; error 45 us; skew 500 ppm
Sep 25 12:19:53 dist-test-slave-clmx krb5kdc[15722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:19:53.231866 16097 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:19:53.232203 16097 webserver.cc:460] Webserver started at http://127.0.98.3:40543/ using document root <none> and password file <none>
I20260925 12:19:53.232383 16097 fs_manager.cc:359] Metadata directory not provided
I20260925 12:19:53.232420 16097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:19:53.232499 16097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:19:53.233369 16097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "ef9694aa8ddb48819d8fbbacf1017fac"
format_stamp: "Formatted at 2026-09-25 12:19:53 on dist-test-slave-clmx"
server_key: "eb32a61c8fbf48a6d2fe661f06704b1d"
server_key_iv: "9507a8cf9db1acb82954e0489a284a60"
server_key_version: "encryptionkey@0"
I20260925 12:19:53.233640 16097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ef9694aa8ddb48819d8fbbacf1017fac"
format_stamp: "Formatted at 2026-09-25 12:19:53 on dist-test-slave-clmx"
server_key: "eb32a61c8fbf48a6d2fe661f06704b1d"
server_key_iv: "9507a8cf9db1acb82954e0489a284a60"
server_key_version: "encryptionkey@0"
I20260925 12:19:53.234905 16097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:19:53.235649 16112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:53.235841 16097 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 12:19:53.235932 16097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "ef9694aa8ddb48819d8fbbacf1017fac"
format_stamp: "Formatted at 2026-09-25 12:19:53 on dist-test-slave-clmx"
server_key: "eb32a61c8fbf48a6d2fe661f06704b1d"
server_key_iv: "9507a8cf9db1acb82954e0489a284a60"
server_key_version: "encryptionkey@0"
I20260925 12:19:53.236014 16097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:19:53.254724 16097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:19:53.256484 16097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:19:53.256635 16097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:19:53.256843 16097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:19:53.257139 16097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:19:53.257191 16097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:53.257241 16097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:19:53.257270 16097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:19:53.262666 16097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33243
I20260925 12:19:53.262717 16225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33243 every 8 connection(s)
I20260925 12:19:53.262977 16097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:19:53.268054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 16097
I20260925 12:19:53.268146 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:19:53.268363 392 external_mini_cluster.cc:1428] Setting key c1188c36a595628cf8d44c352c5a6137
Sep 25 12:19:53 dist-test-slave-clmx krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:19:53.271916 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:53.263656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58637 (local address 127.0.98.62:38775)
0925 12:19:53.263706 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:53.263708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:53.263929 (+ 221us) server_negotiation.cc:407] Connection header received
0925 12:19:53.264354 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:53.264356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:53.264365 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:53.264416 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:53.265230 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.265685 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.266590 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.266837 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.266892 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:53.269187 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:53.269189 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:53.269189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:53.269195 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:53.270581 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:53.271004 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:53.271005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:53.271005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:53.271065 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:53.271272 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:53.271273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:53.271274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:53.271393 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:53.271448 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:53.271770 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 12:19:53.271827 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:19:53.272055 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:53.263608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.3:58637)
0925 12:19:53.263821 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:53.263853 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:53.264280 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:53.264462 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:53.264465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:53.264981 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:19:53.265157 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:53.265161 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.265762 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.265763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:53.266489 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:53.266491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.266908 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.266909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:53.266995 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:53.267471 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:19:53.267479 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:19:53.269108 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:19:53.270673 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:53.270678 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:53.270679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:53.270894 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:53.271127 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:19:53.271128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:19:53.271128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:19:53.271212 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:19:53.271454 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:19:53.271457 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:19:53.271596 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 12:19:53.271748 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 12:19:53.271870 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260925 12:19:53.272521 16226 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38775
I20260925 12:19:53.272593 16226 heartbeater.cc:464] Registering TS with master...
I20260925 12:19:53.272780 16226 heartbeater.cc:511] Master 127.0.98.62:38775 requested a full tablet report, sending...
I20260925 12:19:53.273109 15767 ts_manager.cc:194] Registered new tserver with Master: ef9694aa8ddb48819d8fbbacf1017fac (127.0.98.3:33243)
I20260925 12:19:53.273423 15767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58637
I20260925 12:19:53.280164 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:19:53.287390 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:53.281631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33274 (local address 127.0.98.62:38775)
0925 12:19:53.281672 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:53.281674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:53.281683 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:19:53.281705 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:53.281706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:53.281715 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:53.281791 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:19:53.282312 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.282775 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.283839 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.284047 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.284152 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:53.285253 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:19:53.285254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:19:53.285254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:19:53.285261 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:19:53.286283 (+ 1022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:53.286613 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:53.286614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:53.286614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:53.286670 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:19:53.286871 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:19:53.286872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:19:53.286873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:19:53.287008 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:19:53.287109 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:53.287215 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 12:19:53.287296 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:19:53.288754 15767 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:19:53.290206 15767 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:19:53.296118 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:53.292107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33243 (local address 127.0.0.1:34608)
0925 12:19:53.292332 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:53.292341 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:53.292379 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:53.292938 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:53.292939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:53.292941 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:19:53.292982 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:53.292983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.293835 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.293836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:53.295143 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:53.295145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.295872 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.295873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:53.295936 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:53.295941 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:19:53.295994 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 12:19:53.296035 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":10,"thread_start_us":57,"threads_started":1}
I20260925 12:19:53.296208 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:53.292399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53326 (local address 127.0.98.1:44443)
0925 12:19:53.292545 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:53.292549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:53.292558 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:19:53.292598 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:53.292600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:53.292611 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:53.292703 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:19:53.292828 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.293905 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.294941 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.295621 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.295707 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:53.295779 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:53.296081 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 12:19:53.296124 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
I20260925 12:19:53.296025 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:53.292105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44443 (local address 127.0.0.1:53326)
0925 12:19:53.292260 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:53.292296 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:53.292360 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:53.292717 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:53.292718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:53.292720 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:19:53.292765 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:53.292766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.294041 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.294042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:53.294855 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:53.294858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.295707 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.295707 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:53.295805 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:53.295821 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 12:19:53.295884 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 12:19:53.295933 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":1,"thread_start_us":48,"threads_started":1}
I20260925 12:19:53.296376 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:53.292134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39615 (local address 127.0.0.1:40228)
0925 12:19:53.292445 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:53.292453 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:53.292486 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:53.292678 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:53.292681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:53.292683 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:19:53.292739 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:53.292741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.294010 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.294011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:53.295143 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:53.295145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.296085 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.296086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:53.296160 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:53.296164 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:19:53.296216 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 12:19:53.296258 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":30,"thread_start_us":37,"threads_started":1}
I20260925 12:19:53.296450 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:53.292341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40228 (local address 127.0.98.2:39615)
0925 12:19:53.292507 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:53.292512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:53.292523 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:19:53.292558 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:53.292560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:53.292574 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:53.292647 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:19:53.292823 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.293919 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.295229 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.295953 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.296009 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:53.296055 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:53.296322 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 12:19:53.296371 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
I20260925 12:19:53.296545 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:53.292251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34608 (local address 127.0.98.3:33243)
0925 12:19:53.292783 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:53.292787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:53.292795 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:19:53.292827 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:53.292829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:53.292842 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:53.293053 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:19:53.293113 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.293718 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.295240 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.295793 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.295857 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:53.295911 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:53.296423 (+ 512us) server_negotiation.cc:299] Negotiation successful
0925 12:19:53.296472 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":69,"threads_started":1}
I20260925 12:19:53.297051 16026 tablet_service.cc:1467] Processing CreateTablet for tablet 70b66f1316fa4ab8abb90d5b1d79eee9 (DEFAULT_TABLE table=test-table [id=462e637d7d604ceca0220517ad6d322e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:19:53.297066 15892 tablet_service.cc:1467] Processing CreateTablet for tablet 70b66f1316fa4ab8abb90d5b1d79eee9 (DEFAULT_TABLE table=test-table [id=462e637d7d604ceca0220517ad6d322e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:19:53.297098 16160 tablet_service.cc:1467] Processing CreateTablet for tablet 70b66f1316fa4ab8abb90d5b1d79eee9 (DEFAULT_TABLE table=test-table [id=462e637d7d604ceca0220517ad6d322e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:19:53.297369 16160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b66f1316fa4ab8abb90d5b1d79eee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:19:53.297369 15892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b66f1316fa4ab8abb90d5b1d79eee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:19:53.297386 16026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b66f1316fa4ab8abb90d5b1d79eee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:19:53.299690 16240 tablet_bootstrap.cc:492] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17: Bootstrap starting.
I20260925 12:19:53.300463 16240 tablet_bootstrap.cc:654] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17: Neither blocks nor log segments found. Creating new log.
I20260925 12:19:53.300469 16241 tablet_bootstrap.cc:492] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac: Bootstrap starting.
I20260925 12:19:53.300863 16242 tablet_bootstrap.cc:492] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35: Bootstrap starting.
I20260925 12:19:53.301085 16241 tablet_bootstrap.cc:654] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac: Neither blocks nor log segments found. Creating new log.
I20260925 12:19:53.301071 16240 log.cc:824] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:53.301466 16242 tablet_bootstrap.cc:654] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35: Neither blocks nor log segments found. Creating new log.
I20260925 12:19:53.301564 16241 log.cc:824] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:53.301817 16240 tablet_bootstrap.cc:492] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17: No bootstrap required, opened a new log
I20260925 12:19:53.301900 16240 ts_tablet_manager.cc:1397] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:19:53.302028 16242 log.cc:824] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35: Log is configured to *not* fsync() on all Append() calls
I20260925 12:19:53.302351 16241 tablet_bootstrap.cc:492] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac: No bootstrap required, opened a new log
I20260925 12:19:53.302460 16241 ts_tablet_manager.cc:1397] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:19:53.302613 16242 tablet_bootstrap.cc:492] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35: No bootstrap required, opened a new log
I20260925 12:19:53.302663 16242 ts_tablet_manager.cc:1397] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:19:53.303927 16240 raft_consensus.cc:348] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } }
I20260925 12:19:53.304114 16240 raft_consensus.cc:374] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:19:53.304149 16240 raft_consensus.cc:729] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e6524debb4ecf9571f9b526597f17, State: Initialized, Role: FOLLOWER
I20260925 12:19:53.304286 16240 consensus_queue.cc:260] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } }
I20260925 12:19:53.304242 16242 raft_consensus.cc:348] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } }
I20260925 12:19:53.304428 16242 raft_consensus.cc:374] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:19:53.304463 16242 raft_consensus.cc:729] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c97a4eabb5d4282aae54e4aa821dd35, State: Initialized, Role: FOLLOWER
I20260925 12:19:53.304548 16240 ts_tablet_manager.cc:1428] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 12:19:53.304580 16242 consensus_queue.cc:260] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } }
I20260925 12:19:53.304510 16241 raft_consensus.cc:348] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } }
I20260925 12:19:53.304661 16241 raft_consensus.cc:374] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:19:53.304668 16092 heartbeater.cc:503] Master 127.0.98.62:38775 was elected leader, sending a full tablet report...
I20260925 12:19:53.304698 16241 raft_consensus.cc:729] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef9694aa8ddb48819d8fbbacf1017fac, State: Initialized, Role: FOLLOWER
I20260925 12:19:53.304852 16242 ts_tablet_manager.cc:1428] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:19:53.304890 15958 heartbeater.cc:503] Master 127.0.98.62:38775 was elected leader, sending a full tablet report...
I20260925 12:19:53.304837 16241 consensus_queue.cc:260] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } }
I20260925 12:19:53.305243 16241 ts_tablet_manager.cc:1428] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 12:19:53.305354 16226 heartbeater.cc:503] Master 127.0.98.62:38775 was elected leader, sending a full tablet report...
W20260925 12:19:53.364768 16093 tablet.cc:2369] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 12:19:53.461987 15959 tablet.cc:2369] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:19:53.496568 16248 raft_consensus.cc:482] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:19:53.496668 16248 raft_consensus.cc:504] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } }
I20260925 12:19:53.497032 16248 leader_election.cc:288] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c97a4eabb5d4282aae54e4aa821dd35 (127.0.98.1:44443), 1c0e6524debb4ecf9571f9b526597f17 (127.0.98.2:39615)
I20260925 12:19:53.499915 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:53.497253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39615 (local address 127.0.98.3:51497)
0925 12:19:53.497321 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:53.497334 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:53.497391 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:53.497618 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:53.497619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:53.497620 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:19:53.497693 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:53.497695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.498369 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.498370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:53.499096 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:53.499098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.499658 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.499671 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:53.499741 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:53.499754 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 12:19:53.499799 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 12:19:53.499828 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":28}
I20260925 12:19:53.499927 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:53.497385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51497 (local address 127.0.98.2:39615)
0925 12:19:53.497439 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:53.497441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:53.497450 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:19:53.497478 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:53.497480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:53.497492 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:53.497560 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:19:53.497774 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.498276 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.499184 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.499563 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.499679 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:53.499691 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:53.499820 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 12:19:53.499864 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 12:19:53.499974 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:53.497364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41001 (local address 127.0.98.1:44443)
0925 12:19:53.497419 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:53.497421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:53.497563 (+ 142us) server_negotiation.cc:407] Connection header received
0925 12:19:53.497631 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:53.497632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:53.497644 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:53.497706 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:19:53.497881 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.498354 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.499234 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.499580 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.499707 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:53.499717 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:53.499870 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 12:19:53.499906 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 12:19:53.500010 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:53.497253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44443 (local address 127.0.98.3:41001)
0925 12:19:53.497488 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:53.497498 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:19:53.497553 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:19:53.497722 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:19:53.497740 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:19:53.497742 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:19:53.497813 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:53.497815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.498431 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.498432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:53.499150 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:19:53.499152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.499730 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.499731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:19:53.499780 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:53.499800 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 12:19:53.499855 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 12:19:53.499888 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":81,"thread_start_us":95,"threads_started":1}
I20260925 12:19:53.500229 16046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b66f1316fa4ab8abb90d5b1d79eee9" candidate_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e6524debb4ecf9571f9b526597f17" is_pre_election: true
I20260925 12:19:53.500269 15912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b66f1316fa4ab8abb90d5b1d79eee9" candidate_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" is_pre_election: true
I20260925 12:19:53.500380 16046 raft_consensus.cc:2454] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef9694aa8ddb48819d8fbbacf1017fac in term 0.
I20260925 12:19:53.500380 15912 raft_consensus.cc:2454] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef9694aa8ddb48819d8fbbacf1017fac in term 0.
I20260925 12:19:53.500594 16114 leader_election.cc:302] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e6524debb4ecf9571f9b526597f17, ef9694aa8ddb48819d8fbbacf1017fac; no voters:
I20260925 12:19:53.500780 16248 raft_consensus.cc:2781] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:19:53.500826 16248 raft_consensus.cc:482] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:19:53.500840 16248 raft_consensus.cc:3037] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:19:53.501467 16248 raft_consensus.cc:504] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } }
I20260925 12:19:53.501607 16248 leader_election.cc:288] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [CANDIDATE]: Term 1 election: Requested vote from peers 2c97a4eabb5d4282aae54e4aa821dd35 (127.0.98.1:44443), 1c0e6524debb4ecf9571f9b526597f17 (127.0.98.2:39615)
I20260925 12:19:53.501819 16046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b66f1316fa4ab8abb90d5b1d79eee9" candidate_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e6524debb4ecf9571f9b526597f17"
I20260925 12:19:53.501811 15912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b66f1316fa4ab8abb90d5b1d79eee9" candidate_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c97a4eabb5d4282aae54e4aa821dd35"
I20260925 12:19:53.501886 16046 raft_consensus.cc:3037] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:19:53.501885 15912 raft_consensus.cc:3037] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:19:53.502714 16046 raft_consensus.cc:2454] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef9694aa8ddb48819d8fbbacf1017fac in term 1.
I20260925 12:19:53.502714 15912 raft_consensus.cc:2454] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef9694aa8ddb48819d8fbbacf1017fac in term 1.
I20260925 12:19:53.502887 16115 leader_election.cc:302] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c97a4eabb5d4282aae54e4aa821dd35, ef9694aa8ddb48819d8fbbacf1017fac; no voters:
I20260925 12:19:53.502986 16248 raft_consensus.cc:2781] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:19:53.503139 16248 raft_consensus.cc:686] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [term 1 LEADER]: Becoming Leader. State: Replica: ef9694aa8ddb48819d8fbbacf1017fac, State: Running, Role: LEADER
I20260925 12:19:53.503238 16248 consensus_queue.cc:237] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } }
I20260925 12:19:53.504175 15765 catalog_manager.cc:5495] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac reported cstate change: term changed from 0 to 1, leader changed from <none> to ef9694aa8ddb48819d8fbbacf1017fac (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } health_report { overall_health: UNKNOWN } } }
W20260925 12:19:53.513465 16227 tablet.cc:2369] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:19:53.520030 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:53.517970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34612 (local address 127.0.98.3:33243)
0925 12:19:53.518019 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 12:19:53.518020 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:19:53.518028 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:19:53.518055 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:19:53.518058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:19:53.518078 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:19:53.518132 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:19:53.518233 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.518638 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.519309 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:19:53.519506 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:19:53.519552 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:19:53.519735 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:19:53.519818 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:19:53.519875 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 12:19:53.519907 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 12:19:53.519923 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 12:19:53.521565 16046 raft_consensus.cc:1261] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17 [term 1 FOLLOWER]: Refusing update from remote peer ef9694aa8ddb48819d8fbbacf1017fac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:19:53.521566 15912 raft_consensus.cc:1261] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Refusing update from remote peer ef9694aa8ddb48819d8fbbacf1017fac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:19:53.521953 16250 consensus_queue.cc:1035] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:19:53.522051 16248 consensus_queue.cc:1035] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:19:53.524542 16255 mvcc.cc:205] Tried to move back new op lower bound from 7333227698261766144 to 7333227698190049280. Current Snapshot: MvccSnapshot[applied={T|T < 7333227698261766144}]
I20260925 12:19:53.526319 15767 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:19:53.526405 15767 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:19:53.527041 15767 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9: Sending DeleteTablet for 3 replicas of tablet 70b66f1316fa4ab8abb90d5b1d79eee9
I20260925 12:19:53.527349 16160 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b66f1316fa4ab8abb90d5b1d79eee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:19:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34608
I20260925 12:19:53.527369 16026 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b66f1316fa4ab8abb90d5b1d79eee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:19:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40228
I20260925 12:19:53.527551 15892 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b66f1316fa4ab8abb90d5b1d79eee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:19:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53326
I20260925 12:19:53.527724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 15829
I20260925 12:19:53.528337 16263 tablet_replica.cc:330] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac: stopping tablet replica
I20260925 12:19:53.528420 16263 raft_consensus.cc:2229] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:19:53.528479 16263 raft_consensus.cc:2258] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:19:53.529397 16263 ts_tablet_manager.cc:1905] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:19:53.530797 16263 ts_tablet_manager.cc:1918] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:19:53.530875 16263 log.cc:1196] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/70b66f1316fa4ab8abb90d5b1d79eee9
I20260925 12:19:53.531102 16263 ts_tablet_manager.cc:1939] T 70b66f1316fa4ab8abb90d5b1d79eee9 P ef9694aa8ddb48819d8fbbacf1017fac: Deleting consensus metadata
I20260925 12:19:53.531472 15754 catalog_manager.cc:4841] TS ef9694aa8ddb48819d8fbbacf1017fac (127.0.98.3:33243): tablet 70b66f1316fa4ab8abb90d5b1d79eee9 (table test-table [id=462e637d7d604ceca0220517ad6d322e]) successfully deleted
I20260925 12:19:53.531703 16264 tablet_replica.cc:330] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17: stopping tablet replica
I20260925 12:19:53.531809 16264 raft_consensus.cc:2229] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:19:53.531903 16264 raft_consensus.cc:2258] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:19:53.532793 16264 ts_tablet_manager.cc:1905] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:19:53.534026 16264 ts_tablet_manager.cc:1918] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:19:53.534111 16264 log.cc:1196] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/70b66f1316fa4ab8abb90d5b1d79eee9
I20260925 12:19:53.534348 16264 ts_tablet_manager.cc:1939] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 1c0e6524debb4ecf9571f9b526597f17: Deleting consensus metadata
W20260925 12:19:53.534524 15753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44443 (error 108)
I20260925 12:19:53.534811 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 15963
I20260925 12:19:53.534864 15752 catalog_manager.cc:4841] TS 1c0e6524debb4ecf9571f9b526597f17 (127.0.98.2:39615): tablet 70b66f1316fa4ab8abb90d5b1d79eee9 (table test-table [id=462e637d7d604ceca0220517ad6d322e]) successfully deleted
I20260925 12:19:53.538875 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:19:53.538750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44443 (local address 127.0.0.1:53338)
0925 12:19:53.538792 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 12:19:53.538837 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260925 12:19:53.538997 15753 catalog_manager.cc:4584] TS 2c97a4eabb5d4282aae54e4aa821dd35 (127.0.98.1:44443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 70b66f1316fa4ab8abb90d5b1d79eee9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44443: connect: Connection refused (error 111)
I20260925 12:19:53.540494 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 16097
I20260925 12:19:53.545374 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 15735
I20260925 12:20:13.551397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38775
--webserver_interface=127.0.98.62
--webserver_port=37577
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:20:13.625131 16267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:13.625347 16267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:13.625388 16267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:13.626480 16267 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 12:20:13.626541 16267 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 12:20:13.626564 16267 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 12:20:13.626582 16267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:20:13.626613 16267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:13.626639 16267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:13.626672 16267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:20:13.626700 16267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:20:13.628053 16267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:13.628291 16267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:13.628479 16267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:13.637089 16273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:13.637111 16274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:13.637133 16276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:13.637182 16267 server_base.cc:1034] running on GCE node
I20260925 12:20:13.637503 16267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:13.637797 16267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:13.638940 16267 hybrid_clock.cc:648] HybridClock initialized: now 1790338813638927 us; error 23 us; skew 500 ppm
Sep 25 12:20:13 dist-test-slave-clmx krb5kdc[15722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:13.641459 16267 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:20:13.641945 16267 webserver.cc:460] Webserver started at http://127.0.98.62:37577/ using document root <none> and password file <none>
I20260925 12:20:13.642176 16267 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:13.642256 16267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:13.643708 16267 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:20:13.644326 16283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:13.644486 16267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:20:13.644568 16267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "484a907addb347f89c3c247bd8e796d9"
format_stamp: "Formatted at 2026-09-25 12:19:52 on dist-test-slave-clmx"
server_key: "c65e639ef7b280f6c67c672faf5895f5"
server_key_iv: "3cf47fccc2562dcf12e366aa786d88de"
server_key_version: "encryptionkey@0"
I20260925 12:20:13.644897 16267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:13.670487 16267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:13.672084 16267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:13.672194 16267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:13.675442 16267 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38775
I20260925 12:20:13.675506 16335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38775 every 8 connection(s)
I20260925 12:20:13.675868 16267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:20:13.676003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 16267
I20260925 12:20:13.676307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44443
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38775
--webserver_port=40007
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:20:13.677278 16336 sys_catalog.cc:263] Verifying existing consensus state
I20260925 12:20:13.677860 16336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9: Bootstrap starting.
I20260925 12:20:13.685735 16336 log.cc:824] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:13.687716 16336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:20:13.687978 16336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9: Bootstrap complete.
I20260925 12:20:13.690066 16336 raft_consensus.cc:348] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484a907addb347f89c3c247bd8e796d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38775 } }
I20260925 12:20:13.690289 16336 raft_consensus.cc:729] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484a907addb347f89c3c247bd8e796d9, State: Initialized, Role: FOLLOWER
I20260925 12:20:13.690475 16336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484a907addb347f89c3c247bd8e796d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38775 } }
I20260925 12:20:13.690562 16336 raft_consensus.cc:388] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:20:13.690610 16336 raft_consensus.cc:482] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:20:13.690652 16336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 1 FOLLOWER]: Advancing to term 2
I20260925 12:20:13.691506 16336 raft_consensus.cc:504] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484a907addb347f89c3c247bd8e796d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38775 } }
I20260925 12:20:13.691612 16336 leader_election.cc:302] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 484a907addb347f89c3c247bd8e796d9; no voters:
I20260925 12:20:13.691764 16336 leader_election.cc:288] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 12:20:13.691804 16340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 12:20:13.691974 16340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 2 LEADER]: Becoming Leader. State: Replica: 484a907addb347f89c3c247bd8e796d9, State: Running, Role: LEADER
I20260925 12:20:13.691982 16336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:20:13.692122 16340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484a907addb347f89c3c247bd8e796d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38775 } }
I20260925 12:20:13.692395 16341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "484a907addb347f89c3c247bd8e796d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484a907addb347f89c3c247bd8e796d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38775 } } }
I20260925 12:20:13.692408 16342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 484a907addb347f89c3c247bd8e796d9. Latest consensus state: current_term: 2 leader_uuid: "484a907addb347f89c3c247bd8e796d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484a907addb347f89c3c247bd8e796d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38775 } } }
I20260925 12:20:13.692509 16341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [sys.catalog]: This master's current role is: LEADER
I20260925 12:20:13.692699 16342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [sys.catalog]: This master's current role is: LEADER
I20260925 12:20:13.692981 16345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:20:13.693599 16345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:20:13.693691 16345 catalog_manager.cc:1259] Loaded cluster ID: 31dbbae372d64aaa9bc4f6fa439eb23c
I20260925 12:20:13.693759 16345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:20:13.694888 16345 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:20:13.695047 16345 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9: Loaded TSK: 0
I20260925 12:20:13.712296 16345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9: Generated new TSK 1
I20260925 12:20:13.712378 16345 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 12:20:13.759088 16338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:13.759250 16338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:13.759270 16338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:13.760421 16338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:13.760456 16338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:13.760481 16338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:20:13.761830 16338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40007
--tserver_master_addrs=127.0.98.62:38775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:13.762044 16338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:13.762261 16338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:13.766929 16364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:13.766957 16363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:13.766950 16366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:13.767239 16338 server_base.cc:1034] running on GCE node
I20260925 12:20:13.767428 16338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:13.767689 16338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:13.768859 16338 hybrid_clock.cc:648] HybridClock initialized: now 1790338813768843 us; error 34 us; skew 500 ppm
Sep 25 12:20:13 dist-test-slave-clmx krb5kdc[15722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:13.771304 16338 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:20:13.771847 16338 webserver.cc:460] Webserver started at http://127.0.98.1:40007/ using document root <none> and password file <none>
I20260925 12:20:13.772102 16338 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:13.772174 16338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:13.773687 16338 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:20:13.774407 16373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:13.774581 16338 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:20:13.774654 16338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "2c97a4eabb5d4282aae54e4aa821dd35"
format_stamp: "Formatted at 2026-09-25 12:19:52 on dist-test-slave-clmx"
server_key: "f178376905e9010a59782a68ff64e4b0"
server_key_iv: "60849bfa8ab56febadb62f85fa632a65"
server_key_version: "encryptionkey@0"
I20260925 12:20:13.774971 16338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:13.787345 16338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:13.789465 16338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:13.789599 16338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:13.789824 16338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:13.790198 16380 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 12:20:13.791967 16338 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 12:20:13.792033 16338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 12:20:13.792073 16338 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 12:20:13.792804 16338 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 12:20:13.792862 16338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260925 12:20:13.792915 16380 tablet_bootstrap.cc:492] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35: Bootstrap starting.
I20260925 12:20:13.799804 16338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44443
I20260925 12:20:13.799851 16487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44443 every 8 connection(s)
I20260925 12:20:13.800194 16338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:20:13.802191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 16338
I20260925 12:20:13.802477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39615
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38775
--webserver_port=40879
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:20:13.803184 16380 log.cc:824] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:13.805598 16380 tablet_bootstrap.cc:492] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:20:13.805881 16380 tablet_bootstrap.cc:492] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35: Bootstrap complete.
I20260925 12:20:13.807070 16380 ts_tablet_manager.cc:1397] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
I20260925 12:20:13.808389 16380 raft_consensus.cc:348] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } }
I20260925 12:20:13.808689 16380 raft_consensus.cc:729] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c97a4eabb5d4282aae54e4aa821dd35, State: Initialized, Role: FOLLOWER
I20260925 12:20:13.808838 16380 consensus_queue.cc:260] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } }
I20260925 12:20:13.809078 16380 ts_tablet_manager.cc:1428] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
Sep 25 12:20:13 dist-test-slave-clmx krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:20:13.815965 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:13.800799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.1:41711)
0925 12:20:13.805820 (+ 5021us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:13.805884 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:13.806348 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:13.807048 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:13.807051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:13.807423 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:13.807719 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:13.807725 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:13.809147 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:13.809148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:13.809896 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:13.809897 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:13.810493 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:13.810493 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:13.810626 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:13.811130 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:20:13.811137 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:13.812643 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:13.814431 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:13.814436 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:13.814438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:13.814660 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:13.814903 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:13.814904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:13.814904 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:13.815012 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:13.815349 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:13.815354 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:13.815481 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 12:20:13.815575 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 12:20:13.815750 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4986,"thread_start_us":78,"threads_started":1}
I20260925 12:20:13.815970 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:13.800860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41711 (local address 127.0.98.62:38775)
0925 12:20:13.803437 (+ 2577us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:13.803446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:13.806714 (+ 3268us) server_negotiation.cc:407] Connection header received
0925 12:20:13.806815 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:13.806820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:13.806869 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:13.807008 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:13.807861 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:13.809030 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:13.810013 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:13.810389 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:13.810516 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:13.812740 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:13.812743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:13.812743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:13.812752 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:13.814317 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:13.814745 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:13.814748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:13.814748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:13.814836 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:13.815083 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:13.815084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:13.815084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:13.815277 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:13.815366 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:13.815667 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 12:20:13.815791 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2532,"thread_start_us":64,"threads_started":1}
I20260925 12:20:13.817236 16488 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38775
I20260925 12:20:13.817334 16488 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:13.817488 16488 heartbeater.cc:511] Master 127.0.98.62:38775 requested a full tablet report, sending...
I20260925 12:20:13.818114 16300 ts_manager.cc:194] Registered new tserver with Master: 2c97a4eabb5d4282aae54e4aa821dd35 (127.0.98.1:44443)
I20260925 12:20:13.819216 16300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41711
I20260925 12:20:13.820156 16488 heartbeater.cc:503] Master 127.0.98.62:38775 was elected leader, sending a full tablet report...
Sep 25 12:20:13 dist-test-slave-clmx krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:20:13.826728 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:13.818631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44443 (local address 127.0.0.1:54974)
0925 12:20:13.818964 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:13.818994 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:13.819047 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:13.819340 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:13.819345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:13.819903 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:13.819949 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:13.819951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:13.821116 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:13.821117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:13.821640 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:13.821641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:13.822106 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:13.822109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:13.822177 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:13.822598 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:20:13.822602 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:13.823937 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:13.825639 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:13.825645 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:13.825647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:13.825846 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:13.826062 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:13.826063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:13.826063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:13.826124 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:13.826385 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:13.826391 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:13.826451 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 12:20:13.826525 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 12:20:13.826635 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":70,"threads_started":1}
I20260925 12:20:13.826732 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:13.818841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54974 (local address 127.0.98.1:44443)
0925 12:20:13.819152 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:13.819155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:13.819163 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:20:13.819195 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:13.819196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:13.819208 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:13.819262 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:13.820038 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:13.821033 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:13.821734 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:13.822024 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:13.822097 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:13.824037 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:13.824043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:13.824043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:13.824049 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:13.825538 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:13.825927 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:13.825930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:13.825931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:13.825986 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:13.826187 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:13.826188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:13.826188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:13.826313 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:13.826440 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:13.826574 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 12:20:13.826653 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":56,"threads_started":1}
I20260925 12:20:13.827123 16404 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b66f1316fa4ab8abb90d5b1d79eee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:19:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54974
I20260925 12:20:13.827733 16497 tablet_replica.cc:330] stopping tablet replica
I20260925 12:20:13.827837 16497 raft_consensus.cc:2229] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:20:13.827901 16497 raft_consensus.cc:2258] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:20:13.828183 16497 ts_tablet_manager.cc:1905] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:20:13.830061 16497 ts_tablet_manager.cc:1918] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:20:13.830161 16497 log.cc:1196] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/wal/wals/70b66f1316fa4ab8abb90d5b1d79eee9
I20260925 12:20:13.830387 16497 ts_tablet_manager.cc:1939] T 70b66f1316fa4ab8abb90d5b1d79eee9 P 2c97a4eabb5d4282aae54e4aa821dd35: Deleting consensus metadata
I20260925 12:20:13.830804 16286 catalog_manager.cc:4841] TS 2c97a4eabb5d4282aae54e4aa821dd35 (127.0.98.1:44443): tablet 70b66f1316fa4ab8abb90d5b1d79eee9 (table test-table [id=462e637d7d604ceca0220517ad6d322e]) successfully deleted
W20260925 12:20:13.890447 16493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:13.890645 16493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:13.890683 16493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:13.891920 16493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:13.891974 16493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:13.892016 16493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:20:13.893507 16493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40879
--tserver_master_addrs=127.0.98.62:38775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:13.893740 16493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:13.893944 16493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:13.898307 16503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:13.898317 16502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:13.898365 16505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:13.898653 16493 server_base.cc:1034] running on GCE node
I20260925 12:20:13.898835 16493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:13.899081 16493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:13.900247 16493 hybrid_clock.cc:648] HybridClock initialized: now 1790338813900244 us; error 37 us; skew 500 ppm
Sep 25 12:20:13 dist-test-slave-clmx krb5kdc[15722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:13.902578 16493 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:20:13.902984 16493 webserver.cc:460] Webserver started at http://127.0.98.2:40879/ using document root <none> and password file <none>
I20260925 12:20:13.903190 16493 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:13.903264 16493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:13.904605 16493 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:20:13.905093 16512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:13.905238 16493 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:20:13.905341 16493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "1c0e6524debb4ecf9571f9b526597f17"
format_stamp: "Formatted at 2026-09-25 12:19:53 on dist-test-slave-clmx"
server_key: "f0d753a38533a4cdb53ae9c2e8d4effd"
server_key_iv: "a9409ea16e4d9841c82a0665e5f15efe"
server_key_version: "encryptionkey@0"
I20260925 12:20:13.905632 16493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:13.921299 16493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:13.923404 16493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:13.923574 16493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:13.923822 16493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:13.924095 16493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:13.924149 16493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:13.924187 16493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:13.924216 16493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:13.929714 16493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39615
I20260925 12:20:13.929760 16625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39615 every 8 connection(s)
I20260925 12:20:13.930051 16493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:20:13.934578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 16493
I20260925 12:20:13.934856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33243
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38775
--webserver_port=40543
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 12:20:13 dist-test-slave-clmx krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:20:13.939256 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:13.930684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55593 (local address 127.0.98.62:38775)
0925 12:20:13.930722 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:13.930724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:13.930951 (+ 227us) server_negotiation.cc:407] Connection header received
0925 12:20:13.931450 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:13.931452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:13.931467 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:13.931527 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:13.932434 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:13.932912 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:13.933845 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:13.934049 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:13.934119 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:13.936443 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:13.936444 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:13.936445 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:13.936451 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:13.937785 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:13.938252 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:13.938253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:13.938253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:13.938341 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:13.938640 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:13.938641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:13.938641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:13.938730 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:13.938778 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:13.939133 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 12:20:13.939197 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 12:20:13.939656 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:13.930680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.2:55593)
0925 12:20:13.930843 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:13.930871 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:13.931346 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:13.931590 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:13.931594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:13.932165 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:13.932330 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:13.932333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:13.933002 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:13.933003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:13.933745 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:13.933747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:13.934137 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:13.934138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:13.934267 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:13.934639 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:20:13.934650 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:13.936345 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:13.937867 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:13.937873 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:13.937875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:13.938182 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:13.938429 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:13.938430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:13.938431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:13.938570 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:13.938811 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:13.938817 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:13.938993 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 12:20:13.939109 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 12:20:13.939279 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260925 12:20:13.940157 16626 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38775
I20260925 12:20:13.940259 16626 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:13.940416 16626 heartbeater.cc:511] Master 127.0.98.62:38775 requested a full tablet report, sending...
I20260925 12:20:13.940776 16300 ts_manager.cc:194] Registered new tserver with Master: 1c0e6524debb4ecf9571f9b526597f17 (127.0.98.2:39615)
I20260925 12:20:13.941124 16300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55593
W20260925 12:20:14.012470 16629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:14.012650 16629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:14.012671 16629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:14.013767 16629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:14.013803 16629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:14.013827 16629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:20:14.015151 16629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40543
--tserver_master_addrs=127.0.98.62:38775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:14.015348 16629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:14.015586 16629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:14.020033 16635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:14.020073 16634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:14.020035 16637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:14.020282 16629 server_base.cc:1034] running on GCE node
I20260925 12:20:14.020495 16629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:14.020761 16629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:14.021950 16629 hybrid_clock.cc:648] HybridClock initialized: now 1790338814021938 us; error 27 us; skew 500 ppm
Sep 25 12:20:14 dist-test-slave-clmx krb5kdc[15722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:14.024420 16629 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:20:14.024860 16629 webserver.cc:460] Webserver started at http://127.0.98.3:40543/ using document root <none> and password file <none>
I20260925 12:20:14.025072 16629 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:14.025125 16629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:14.026363 16629 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:20:14.026911 16644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:14.027077 16629 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:20:14.027158 16629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "ef9694aa8ddb48819d8fbbacf1017fac"
format_stamp: "Formatted at 2026-09-25 12:19:53 on dist-test-slave-clmx"
server_key: "eb32a61c8fbf48a6d2fe661f06704b1d"
server_key_iv: "9507a8cf9db1acb82954e0489a284a60"
server_key_version: "encryptionkey@0"
I20260925 12:20:14.027420 16629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:14.038528 16629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:14.040313 16629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:14.040408 16629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:14.040563 16629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:14.040773 16629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:14.040798 16629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:14.040817 16629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:14.040828 16629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:14.045985 16629 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33243
I20260925 12:20:14.046056 16757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33243 every 8 connection(s)
I20260925 12:20:14.046344 16629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:20:14.049933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 16629
Sep 25 12:20:14 dist-test-slave-clmx krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:20:14.055130 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:14.047011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42831 (local address 127.0.98.62:38775)
0925 12:20:14.047049 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:14.047051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:14.047249 (+ 198us) server_negotiation.cc:407] Connection header received
0925 12:20:14.047721 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:14.047722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:14.047732 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:14.047790 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:14.048539 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.048970 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.049795 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.050066 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.050137 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:14.052505 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:14.052506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:14.052507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:14.052513 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:14.053903 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:14.054279 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:14.054281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:14.054281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:14.054334 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:14.054565 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:14.054566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:14.054567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:14.054661 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:14.054735 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:14.054975 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 12:20:14.055048 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 12:20:14.055299 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:14.046977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.3:42831)
0925 12:20:14.047140 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:14.047175 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:14.047592 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:14.047853 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:14.047855 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:14.048312 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:14.048472 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:14.048475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.049040 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.049041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:14.049723 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:14.049725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.050155 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.050156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:14.050289 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:14.050767 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:20:14.050776 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:14.052417 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:14.053998 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:14.054003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:14.054005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:14.054197 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:14.054396 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:14.054397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:14.054397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:14.054496 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:14.054733 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:14.054739 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:14.054856 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 12:20:14.054955 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 12:20:14.055084 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
I20260925 12:20:14.055940 16758 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38775
I20260925 12:20:14.056030 16758 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:14.056209 16758 heartbeater.cc:511] Master 127.0.98.62:38775 requested a full tablet report, sending...
I20260925 12:20:14.056608 16300 ts_manager.cc:194] Registered new tserver with Master: ef9694aa8ddb48819d8fbbacf1017fac (127.0.98.3:33243)
I20260925 12:20:14.056895 16761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:14.050297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50422 (local address 127.0.98.62:38775)
0925 12:20:14.050394 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:14.050398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:14.050530 (+ 132us) server_negotiation.cc:407] Connection header received
0925 12:20:14.050586 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:14.050587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:14.050598 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:14.050654 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:14.051375 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.052119 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.052916 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.053130 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.053206 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:14.054193 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:14.054195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:14.054195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:14.054201 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:14.055738 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:14.056161 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:14.056162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:14.056162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:14.056210 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:14.056408 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:14.056409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:14.056409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:14.056540 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:14.056705 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:14.056743 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 12:20:14.056812 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":64,"threads_started":1}
I20260925 12:20:14.057080 16300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42831
I20260925 12:20:14.057533 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:20:14.059710 16761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:14.057826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.0.98.62:38775)
0925 12:20:14.057866 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:14.057868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:14.057940 (+ 72us) server_negotiation.cc:407] Connection header received
0925 12:20:14.057989 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:14.057989 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:14.058011 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:14.058064 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:20:14.058123 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.058521 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.059165 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.059337 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.059381 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:14.059491 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:20:14.059522 (+ 31us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 12:20:14.059655 (+ 133us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50434: token expired
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 12:20:14.059758 16761 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50434: token expired
W20260925 12:20:14.059825 16230 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:38775)
I20260925 12:20:14.064747 16761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:14.060034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.0.98.62:38775)
0925 12:20:14.060064 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:14.060066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:14.060104 (+ 38us) server_negotiation.cc:407] Connection header received
0925 12:20:14.060146 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:14.060146 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:14.060152 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:14.060195 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:14.060565 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.060947 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.061575 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.061727 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.061769 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:14.062761 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:14.062761 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:14.062762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:14.062766 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:14.063939 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:14.064211 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:14.064212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:14.064212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:14.064255 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:14.064397 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:14.064398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:14.064398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:14.064477 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:14.064537 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:14.064604 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 12:20:14.064655 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 12:20:14.065680 16300 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:20:14.067075 16300 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:20:14.069080 16404 tablet_service.cc:1467] Processing CreateTablet for tablet 80814af23d614ef9aa42426d1f0797fe (DEFAULT_TABLE table=test-table [id=f541583bc63244f7a71d3993fc380511]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:14.069275 16404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80814af23d614ef9aa42426d1f0797fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:14.070866 16380 tablet_bootstrap.cc:492] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35: Bootstrap starting.
I20260925 12:20:14.071287 16380 tablet_bootstrap.cc:654] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:14.071942 16380 tablet_bootstrap.cc:492] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35: No bootstrap required, opened a new log
I20260925 12:20:14.071985 16380 ts_tablet_manager.cc:1397] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:20:14.072304 16380 raft_consensus.cc:348] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260925 12:20:14.072389 16380 raft_consensus.cc:374] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:14.072435 16380 raft_consensus.cc:729] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c97a4eabb5d4282aae54e4aa821dd35, State: Initialized, Role: FOLLOWER
I20260925 12:20:14.072479 16380 consensus_queue.cc:260] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260925 12:20:14.072579 16380 ts_tablet_manager.cc:1428] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:20:14.072683 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:14.068796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39615 (local address 127.0.0.1:43518)
0925 12:20:14.068824 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:14.068834 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:14.068870 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:14.069275 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:14.069276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:14.069277 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:14.069315 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:14.069316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.070252 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.070253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:14.071349 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:14.071351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.072410 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.072411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:14.072491 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:14.072510 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 12:20:14.072566 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 12:20:14.072611 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260925 12:20:14.072911 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:14.068951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.0.98.2:39615)
0925 12:20:14.069119 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:14.069123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:14.069134 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:20:14.069171 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:14.069173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:14.069188 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:14.069267 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:14.069368 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.070172 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.071450 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.072325 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.072392 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:14.072458 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:14.072774 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 12:20:14.072820 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260925 12:20:14.073539 16560 tablet_service.cc:1467] Processing CreateTablet for tablet 80814af23d614ef9aa42426d1f0797fe (DEFAULT_TABLE table=test-table [id=f541583bc63244f7a71d3993fc380511]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:14.073853 16560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80814af23d614ef9aa42426d1f0797fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:14.074307 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:14.068869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33243 (local address 127.0.0.1:39058)
0925 12:20:14.069122 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:14.069132 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:14.069178 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:14.071034 (+ 1856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:14.071035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:14.071036 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:14.071082 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:14.071083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.072233 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.072233 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:14.073351 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:14.073353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.074148 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.074149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:14.074190 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:14.074192 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:20:14.074238 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 12:20:14.074264 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":81,"threads_started":1}
I20260925 12:20:14.074363 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:14.069033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.0.98.3:33243)
0925 12:20:14.070899 (+ 1866us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:14.070903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:14.070913 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:20:14.070944 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:14.070945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:14.070958 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:14.071034 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:14.071152 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.072129 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.073425 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.074075 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.074149 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:14.074200 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:14.074249 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 12:20:14.074281 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1837,"thread_start_us":71,"threads_started":1}
I20260925 12:20:14.075028 16692 tablet_service.cc:1467] Processing CreateTablet for tablet 80814af23d614ef9aa42426d1f0797fe (DEFAULT_TABLE table=test-table [id=f541583bc63244f7a71d3993fc380511]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:14.075302 16692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80814af23d614ef9aa42426d1f0797fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:14.077158 16768 tablet_bootstrap.cc:492] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17: Bootstrap starting.
I20260925 12:20:14.077863 16768 tablet_bootstrap.cc:654] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:14.078429 16768 log.cc:824] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:14.078513 16769 tablet_bootstrap.cc:492] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac: Bootstrap starting.
I20260925 12:20:14.079114 16769 tablet_bootstrap.cc:654] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:14.079140 16768 tablet_bootstrap.cc:492] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17: No bootstrap required, opened a new log
I20260925 12:20:14.079187 16768 ts_tablet_manager.cc:1397] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:20:14.079715 16769 log.cc:824] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:14.080446 16769 tablet_bootstrap.cc:492] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac: No bootstrap required, opened a new log
I20260925 12:20:14.080533 16769 ts_tablet_manager.cc:1397] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 12:20:14.080973 16768 raft_consensus.cc:348] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260925 12:20:14.081112 16768 raft_consensus.cc:374] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:14.081154 16768 raft_consensus.cc:729] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e6524debb4ecf9571f9b526597f17, State: Initialized, Role: FOLLOWER
I20260925 12:20:14.081260 16768 consensus_queue.cc:260] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260925 12:20:14.081533 16768 ts_tablet_manager.cc:1428] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 12:20:14.081676 16626 heartbeater.cc:503] Master 127.0.98.62:38775 was elected leader, sending a full tablet report...
I20260925 12:20:14.082191 16769 raft_consensus.cc:348] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260925 12:20:14.082371 16769 raft_consensus.cc:374] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:14.082438 16769 raft_consensus.cc:729] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef9694aa8ddb48819d8fbbacf1017fac, State: Initialized, Role: FOLLOWER
I20260925 12:20:14.082536 16769 consensus_queue.cc:260] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260925 12:20:14.082772 16769 ts_tablet_manager.cc:1428] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 12:20:14.082826 16758 heartbeater.cc:503] Master 127.0.98.62:38775 was elected leader, sending a full tablet report...
I20260925 12:20:14.092456 16772 raft_consensus.cc:482] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:20:14.092523 16772 raft_consensus.cc:504] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260925 12:20:14.092811 16772 leader_election.cc:288] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef9694aa8ddb48819d8fbbacf1017fac (127.0.98.3:33243), 2c97a4eabb5d4282aae54e4aa821dd35 (127.0.98.1:44443)
I20260925 12:20:14.095178 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:14.092949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33243 (local address 127.0.98.2:33789)
0925 12:20:14.092989 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:14.093001 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:14.093045 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:14.093181 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:14.093182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:14.093183 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:14.093238 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:14.093239 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.093779 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.093780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:14.094411 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:14.094413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.094914 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.094915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:14.094991 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:14.094994 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:20:14.095072 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 12:20:14.095114 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 12:20:14.095187 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:14.093043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33789 (local address 127.0.98.3:33243)
0925 12:20:14.093076 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:14.093078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:14.093085 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:20:14.093103 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:14.093104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:14.093111 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:14.093181 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:14.093313 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.093710 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.094505 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.094824 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.094915 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:14.094923 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:14.095092 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 12:20:14.095136 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:20:14.095409 16712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80814af23d614ef9aa42426d1f0797fe" candidate_uuid: "1c0e6524debb4ecf9571f9b526597f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" is_pre_election: true
I20260925 12:20:14.095548 16712 raft_consensus.cc:2454] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e6524debb4ecf9571f9b526597f17 in term 0.
I20260925 12:20:14.095750 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:14.092958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44443 (local address 127.0.98.2:57383)
0925 12:20:14.093144 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:14.093151 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:14.093257 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:14.093431 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:14.093432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:14.093433 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:14.093494 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:14.093496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.094095 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.094096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:14.094751 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:14.094752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.095472 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.095473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:14.095545 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:14.095549 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:20:14.095609 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 12:20:14.095665 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":10,"thread_start_us":75,"threads_started":1}
I20260925 12:20:14.095777 16516 leader_election.cc:302] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e6524debb4ecf9571f9b526597f17, ef9694aa8ddb48819d8fbbacf1017fac; no voters:
I20260925 12:20:14.095779 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:14.093050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57383 (local address 127.0.98.1:44443)
0925 12:20:14.093098 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:14.093100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:14.093235 (+ 135us) server_negotiation.cc:407] Connection header received
0925 12:20:14.093320 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:14.093321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:14.093339 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:14.093426 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:14.093564 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.094008 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.094832 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.095396 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.095460 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:14.095532 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:14.095664 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 12:20:14.095707 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 12:20:14.095875 16772 raft_consensus.cc:2781] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:20:14.095914 16772 raft_consensus.cc:482] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:20:14.095937 16772 raft_consensus.cc:3037] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:14.096058 16442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80814af23d614ef9aa42426d1f0797fe" candidate_uuid: "1c0e6524debb4ecf9571f9b526597f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" is_pre_election: true
I20260925 12:20:14.096201 16442 raft_consensus.cc:2454] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e6524debb4ecf9571f9b526597f17 in term 0.
I20260925 12:20:14.096647 16772 raft_consensus.cc:504] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260925 12:20:14.096823 16772 leader_election.cc:288] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [CANDIDATE]: Term 1 election: Requested vote from peers ef9694aa8ddb48819d8fbbacf1017fac (127.0.98.3:33243), 2c97a4eabb5d4282aae54e4aa821dd35 (127.0.98.1:44443)
I20260925 12:20:14.096891 16712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80814af23d614ef9aa42426d1f0797fe" candidate_uuid: "1c0e6524debb4ecf9571f9b526597f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef9694aa8ddb48819d8fbbacf1017fac"
I20260925 12:20:14.096948 16712 raft_consensus.cc:3037] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:14.096956 16442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80814af23d614ef9aa42426d1f0797fe" candidate_uuid: "1c0e6524debb4ecf9571f9b526597f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c97a4eabb5d4282aae54e4aa821dd35"
I20260925 12:20:14.097028 16442 raft_consensus.cc:3037] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:14.097630 16712 raft_consensus.cc:2454] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e6524debb4ecf9571f9b526597f17 in term 1.
I20260925 12:20:14.097733 16442 raft_consensus.cc:2454] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e6524debb4ecf9571f9b526597f17 in term 1.
I20260925 12:20:14.097797 16516 leader_election.cc:302] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e6524debb4ecf9571f9b526597f17, ef9694aa8ddb48819d8fbbacf1017fac; no voters:
I20260925 12:20:14.097885 16772 raft_consensus.cc:2781] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:20:14.097997 16772 raft_consensus.cc:686] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e6524debb4ecf9571f9b526597f17, State: Running, Role: LEADER
I20260925 12:20:14.098088 16772 consensus_queue.cc:237] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260925 12:20:14.098781 16300 catalog_manager.cc:5495] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e6524debb4ecf9571f9b526597f17 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1c0e6524debb4ecf9571f9b526597f17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } health_report { overall_health: UNKNOWN } } }
I20260925 12:20:14.108541 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:14.106537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43530 (local address 127.0.98.2:39615)
0925 12:20:14.106572 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:14.106574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:14.106583 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:20:14.106660 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:14.106661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:14.106674 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:14.106727 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:20:14.106808 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.107227 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.107943 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:14.108104 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:14.108153 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:14.108258 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:20:14.108333 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:20:14.108386 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:14.108427 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 12:20:14.108442 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 12:20:14.109999 16442 raft_consensus.cc:1261] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e6524debb4ecf9571f9b526597f17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:20:14.109999 16712 raft_consensus.cc:1261] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac [term 1 FOLLOWER]: Refusing update from remote peer 1c0e6524debb4ecf9571f9b526597f17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:20:14.110347 16772 consensus_queue.cc:1035] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:20:14.110507 16775 consensus_queue.cc:1035] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:20:14.112703 16780 mvcc.cc:205] Tried to move back new op lower bound from 7333227782592249856 to 7333227782546358272. Current Snapshot: MvccSnapshot[applied={T|T < 7333227782592249856}]
I20260925 12:20:14.114733 16299 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:20:14.114825 16299 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:20:14.115483 16299 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9: Sending DeleteTablet for 3 replicas of tablet 80814af23d614ef9aa42426d1f0797fe
I20260925 12:20:14.115832 16404 tablet_service.cc:1514] Processing DeleteTablet for tablet 80814af23d614ef9aa42426d1f0797fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:20:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54974
I20260925 12:20:14.115867 16560 tablet_service.cc:1514] Processing DeleteTablet for tablet 80814af23d614ef9aa42426d1f0797fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:20:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43518
I20260925 12:20:14.115993 16692 tablet_service.cc:1514] Processing DeleteTablet for tablet 80814af23d614ef9aa42426d1f0797fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:20:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39058
I20260925 12:20:14.116117 16787 tablet_replica.cc:330] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17: stopping tablet replica
I20260925 12:20:14.116160 16497 tablet_replica.cc:330] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35: stopping tablet replica
I20260925 12:20:14.116186 16788 tablet_replica.cc:330] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac: stopping tablet replica
I20260925 12:20:14.116226 16497 raft_consensus.cc:2229] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:20:14.116261 16788 raft_consensus.cc:2229] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:20:14.116285 16497 raft_consensus.cc:2258] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:20:14.116357 16788 raft_consensus.cc:2258] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:20:14.116454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 16338
I20260925 12:20:14.116371 16787 raft_consensus.cc:2229] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:20:14.116671 16787 raft_consensus.cc:2258] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:20:14.117164 16787 ts_tablet_manager.cc:1905] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:20:14.117197 16788 ts_tablet_manager.cc:1905] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:20:14.118821 16787 ts_tablet_manager.cc:1918] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:20:14.118949 16787 log.cc:1196] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/80814af23d614ef9aa42426d1f0797fe
I20260925 12:20:14.118948 16788 ts_tablet_manager.cc:1918] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:20:14.119004 16788 log.cc:1196] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/80814af23d614ef9aa42426d1f0797fe
I20260925 12:20:14.119302 16788 ts_tablet_manager.cc:1939] T 80814af23d614ef9aa42426d1f0797fe P ef9694aa8ddb48819d8fbbacf1017fac: Deleting consensus metadata
I20260925 12:20:14.119390 16787 ts_tablet_manager.cc:1939] T 80814af23d614ef9aa42426d1f0797fe P 1c0e6524debb4ecf9571f9b526597f17: Deleting consensus metadata
I20260925 12:20:14.119665 16287 catalog_manager.cc:4841] TS ef9694aa8ddb48819d8fbbacf1017fac (127.0.98.3:33243): tablet 80814af23d614ef9aa42426d1f0797fe (table test-table [id=f541583bc63244f7a71d3993fc380511]) successfully deleted
I20260925 12:20:14.119788 16285 catalog_manager.cc:4841] TS 1c0e6524debb4ecf9571f9b526597f17 (127.0.98.2:39615): tablet 80814af23d614ef9aa42426d1f0797fe (table test-table [id=f541583bc63244f7a71d3993fc380511]) successfully deleted
W20260925 12:20:14.122661 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44443 (error 108)
I20260925 12:20:14.123009 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 16493
I20260925 12:20:14.123091 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:14.122989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44443 (local address 127.0.0.1:54984)
0925 12:20:14.123027 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:14.123060 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260925 12:20:14.123610 16286 catalog_manager.cc:4584] TS 2c97a4eabb5d4282aae54e4aa821dd35 (127.0.98.1:44443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80814af23d614ef9aa42426d1f0797fe: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44443: connect: Connection refused (error 111)
I20260925 12:20:14.127332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 16629
I20260925 12:20:14.130957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 16267
I20260925 12:20:34.135463 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38775
--webserver_interface=127.0.98.62
--webserver_port=37577
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:20:34.211804 16790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:34.211977 16790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:34.211997 16790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:34.213069 16790 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 12:20:34.213114 16790 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 12:20:34.213125 16790 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 12:20:34.213136 16790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:20:34.213181 16790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:34.213200 16790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:34.213223 16790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:20:34.213248 16790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:20:34.214636 16790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:34.214826 16790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:34.215025 16790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:34.219774 16795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:34.219782 16798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:34.219823 16790 server_base.cc:1034] running on GCE node
W20260925 12:20:34.219786 16796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:34.220222 16790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:34.220435 16790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:34.221570 16790 hybrid_clock.cc:648] HybridClock initialized: now 1790338834221550 us; error 34 us; skew 500 ppm
Sep 25 12:20:34 dist-test-slave-clmx krb5kdc[15722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:34.223798 16790 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:20:34.224207 16790 webserver.cc:460] Webserver started at http://127.0.98.62:37577/ using document root <none> and password file <none>
I20260925 12:20:34.224419 16790 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:34.224462 16790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:34.225663 16790 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:20:34.226264 16805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:34.226423 16790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:20:34.226567 16790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "484a907addb347f89c3c247bd8e796d9"
format_stamp: "Formatted at 2026-09-25 12:19:52 on dist-test-slave-clmx"
server_key: "c65e639ef7b280f6c67c672faf5895f5"
server_key_iv: "3cf47fccc2562dcf12e366aa786d88de"
server_key_version: "encryptionkey@0"
I20260925 12:20:34.226853 16790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:34.238112 16790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:34.239801 16790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:34.239940 16790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:34.243819 16790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38775
I20260925 12:20:34.243846 16857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38775 every 8 connection(s)
I20260925 12:20:34.244196 16790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:20:34.245424 16858 sys_catalog.cc:263] Verifying existing consensus state
I20260925 12:20:34.245945 16858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9: Bootstrap starting.
I20260925 12:20:34.249756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 16790
I20260925 12:20:34.249998 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44443
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38775
--webserver_port=40007
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:20:34.253974 16858 log.cc:824] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:34.256022 16858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:20:34.256274 16858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9: Bootstrap complete.
I20260925 12:20:34.258327 16858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484a907addb347f89c3c247bd8e796d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38775 } }
I20260925 12:20:34.258524 16858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484a907addb347f89c3c247bd8e796d9, State: Initialized, Role: FOLLOWER
I20260925 12:20:34.258678 16858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484a907addb347f89c3c247bd8e796d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38775 } }
I20260925 12:20:34.258734 16858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:20:34.258790 16858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:20:34.258829 16858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 2 FOLLOWER]: Advancing to term 3
I20260925 12:20:34.259789 16858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484a907addb347f89c3c247bd8e796d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38775 } }
I20260925 12:20:34.259882 16858 leader_election.cc:302] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 484a907addb347f89c3c247bd8e796d9; no voters:
I20260925 12:20:34.260048 16858 leader_election.cc:288] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 12:20:34.260079 16862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 12:20:34.260195 16862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [term 3 LEADER]: Becoming Leader. State: Replica: 484a907addb347f89c3c247bd8e796d9, State: Running, Role: LEADER
I20260925 12:20:34.260267 16858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:20:34.260267 16862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484a907addb347f89c3c247bd8e796d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38775 } }
I20260925 12:20:34.260556 16864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 484a907addb347f89c3c247bd8e796d9. Latest consensus state: current_term: 3 leader_uuid: "484a907addb347f89c3c247bd8e796d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484a907addb347f89c3c247bd8e796d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38775 } } }
I20260925 12:20:34.260717 16864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [sys.catalog]: This master's current role is: LEADER
I20260925 12:20:34.260552 16863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "484a907addb347f89c3c247bd8e796d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484a907addb347f89c3c247bd8e796d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38775 } } }
I20260925 12:20:34.260839 16863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9 [sys.catalog]: This master's current role is: LEADER
I20260925 12:20:34.260921 16870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:20:34.261651 16870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:20:34.261749 16870 catalog_manager.cc:1259] Loaded cluster ID: 31dbbae372d64aaa9bc4f6fa439eb23c
I20260925 12:20:34.261781 16870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:20:34.262794 16870 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:20:34.262945 16870 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9: Loaded TSK: 0
I20260925 12:20:34.262987 16870 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 484a907addb347f89c3c247bd8e796d9: Loaded TSK: 1
F20260925 12:20:34.263274 16870 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc2d3f3b9ad google::LogMessage::Fail() at ??:0
@ 0x7fc2d3f3e88a google::LogMessage::SendToLog() at ??:0
@ 0x7fc2d3f3b4de google::LogMessage::Flush() at ??:0
@ 0x7fc2d3f3ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc2d66a8104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc2d66a5d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc2dbe951cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc2dbe9548e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc2dbe95594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fc2dbe5bc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fc2dbe65ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc2d5e090d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc2d5dfe954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc2d6c626db start_thread at ??:0
@ 0x7fc2d2a3961f clone at ??:0
W20260925 12:20:34.352470 16860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:34.352732 16860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:34.352767 16860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:34.354458 16860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:34.354533 16860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:34.354573 16860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:20:34.356462 16860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40007
--tserver_master_addrs=127.0.98.62:38775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:34.356679 16860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:34.356890 16860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:34.361676 16900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:34.361670 16899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:34.361887 16860 server_base.cc:1034] running on GCE node
W20260925 12:20:34.361667 16902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:34.362161 16860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:34.362403 16860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:34.363559 16860 hybrid_clock.cc:648] HybridClock initialized: now 1790338834363547 us; error 29 us; skew 500 ppm
Sep 25 12:20:34 dist-test-slave-clmx krb5kdc[15722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:34.366045 16860 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:20:34.366493 16860 webserver.cc:460] Webserver started at http://127.0.98.1:40007/ using document root <none> and password file <none>
I20260925 12:20:34.366699 16860 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:34.366762 16860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:34.368074 16860 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 12:20:34.368772 16909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:34.368934 16860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 12:20:34.369040 16860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "2c97a4eabb5d4282aae54e4aa821dd35"
format_stamp: "Formatted at 2026-09-25 12:19:52 on dist-test-slave-clmx"
server_key: "f178376905e9010a59782a68ff64e4b0"
server_key_iv: "60849bfa8ab56febadb62f85fa632a65"
server_key_version: "encryptionkey@0"
I20260925 12:20:34.369334 16860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:34.382133 16860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:34.384302 16860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:34.384472 16860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:34.384708 16860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:34.385314 16916 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 12:20:34.387224 16860 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 12:20:34.387272 16860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 12:20:34.387317 16860 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 12:20:34.388144 16860 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 12:20:34.388211 16860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 12:20:34.388240 16916 tablet_bootstrap.cc:492] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35: Bootstrap starting.
I20260925 12:20:34.394907 16860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44443
I20260925 12:20:34.395226 16860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:20:34.398751 16916 log.cc:824] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:34.399082 17023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44443 every 8 connection(s)
I20260925 12:20:34.400784 16916 tablet_bootstrap.cc:492] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:20:34.401041 16916 tablet_bootstrap.cc:492] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35: Bootstrap complete.
I20260925 12:20:34.401893 16916 ts_tablet_manager.cc:1397] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20260925 12:20:34.402808 16916 raft_consensus.cc:348] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260925 12:20:34.402995 16916 raft_consensus.cc:729] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c97a4eabb5d4282aae54e4aa821dd35, State: Initialized, Role: FOLLOWER
I20260925 12:20:34.403115 16916 consensus_queue.cc:260] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260925 12:20:34.403358 16916 ts_tablet_manager.cc:1428] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20260925 12:20:34.404731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 16860
I20260925 12:20:34.404979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39615
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38775
--webserver_port=40879
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:20:34.494751 17026 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:20:34.395814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.1:41215)
0925 12:20:34.395970 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:34.396007 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:34.396405 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:34.494540 (+ 98135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":96,"threads_started":1}
W20260925 12:20:34.495023 17024 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38775 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38775: Client connection negotiation failed: client connection to 127.0.98.62:38775: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 12:20:34.495362 17026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:34.495252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.1:57661)
0925 12:20:34.495303 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:34.495338 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 12:20:34.495740 17026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:34.495663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.1:52323)
0925 12:20:34.495695 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:34.495724 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260925 12:20:34.495858 17024 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 12:20:34.511503 17029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:34.511740 17029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:34.511790 17029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:34.512902 17029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:34.512943 17029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:34.512996 17029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:20:34.514371 17029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40879
--tserver_master_addrs=127.0.98.62:38775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:34.514637 17029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:34.514859 17029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:34.519588 17034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:34.519634 17037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:34.519662 17029 server_base.cc:1034] running on GCE node
W20260925 12:20:34.519634 17035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:34.519969 17029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:34.520190 17029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:34.521344 17029 hybrid_clock.cc:648] HybridClock initialized: now 1790338834521345 us; error 39 us; skew 500 ppm
Sep 25 12:20:34 dist-test-slave-clmx krb5kdc[15722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:34.523761 17029 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:20:34.524173 17029 webserver.cc:460] Webserver started at http://127.0.98.2:40879/ using document root <none> and password file <none>
I20260925 12:20:34.524398 17029 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:34.524463 17029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:34.525661 17029 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:20:34.526178 17044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:34.526326 17029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:20:34.526417 17029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "1c0e6524debb4ecf9571f9b526597f17"
format_stamp: "Formatted at 2026-09-25 12:19:53 on dist-test-slave-clmx"
server_key: "f0d753a38533a4cdb53ae9c2e8d4effd"
server_key_iv: "a9409ea16e4d9841c82a0665e5f15efe"
server_key_version: "encryptionkey@0"
I20260925 12:20:34.526716 17029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:34.544586 17029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:34.546636 17029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:34.546772 17029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:34.546965 17029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:34.547219 17029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:34.547267 17029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:34.547303 17029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:34.547328 17029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:34.553064 17029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39615
I20260925 12:20:34.553158 17157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39615 every 8 connection(s)
I20260925 12:20:34.553444 17029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:20:34.554386 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:34.554076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.2:41205)
0925 12:20:34.554206 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:34.554264 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
W20260925 12:20:34.554801 17158 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38775 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38775: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
I20260925 12:20:34.555028 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:34.554952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.2:35971)
0925 12:20:34.554989 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:34.555013 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 12:20:34.555277 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:34.555209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.2:43235)
0925 12:20:34.555245 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:34.555264 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 12:20:34.555397 17158 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 12:20:34.560508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 17029
I20260925 12:20:34.560707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33243
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38775
--webserver_port=40543
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:20:34.635358 17161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:34.635516 17161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:34.635536 17161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:34.636623 17161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:34.636662 17161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:34.636686 17161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:20:34.638049 17161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40543
--tserver_master_addrs=127.0.98.62:38775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:34.638270 17161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:34.638477 17161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:34.642942 17169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:34.642952 17166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:34.643011 17161 server_base.cc:1034] running on GCE node
W20260925 12:20:34.642987 17167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:34.643357 17161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:34.643573 17161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:34.644742 17161 hybrid_clock.cc:648] HybridClock initialized: now 1790338834644724 us; error 33 us; skew 500 ppm
Sep 25 12:20:34 dist-test-slave-clmx krb5kdc[15722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:34.646914 17161 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:20:34.647248 17161 webserver.cc:460] Webserver started at http://127.0.98.3:40543/ using document root <none> and password file <none>
I20260925 12:20:34.647401 17161 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:34.647439 17161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:34.648641 17161 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:20:34.649196 17176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:34.649346 17161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:34.649425 17161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "ef9694aa8ddb48819d8fbbacf1017fac"
format_stamp: "Formatted at 2026-09-25 12:19:53 on dist-test-slave-clmx"
server_key: "eb32a61c8fbf48a6d2fe661f06704b1d"
server_key_iv: "9507a8cf9db1acb82954e0489a284a60"
server_key_version: "encryptionkey@0"
I20260925 12:20:34.649688 17161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:34.662899 17161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:34.665050 17161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:34.665145 17161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:34.665292 17161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:34.665490 17161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:34.665524 17161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:34.665544 17161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:34.665560 17161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:34.670799 17161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33243
I20260925 12:20:34.670884 17289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33243 every 8 connection(s)
I20260925 12:20:34.671166 17161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:20:34.672160 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:34.671795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.3:44439)
0925 12:20:34.671932 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:34.671992 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
W20260925 12:20:34.672544 17290 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38775 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38775: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
I20260925 12:20:34.672816 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:34.672722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.3:49567)
0925 12:20:34.672768 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:34.672796 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 12:20:34.673053 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:34.672999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.3:42939)
0925 12:20:34.673026 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:34.673042 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260925 12:20:34.673161 17290 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 12:20:34.674947 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 17161
I20260925 12:20:35.496363 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:35.496153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.1:43445)
0925 12:20:35.496266 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:35.496299 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260925 12:20:35.555892 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:35.555715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.2:60843)
0925 12:20:35.555813 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:35.555851 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":53,"threads_started":1}
I20260925 12:20:35.673584 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:35.673415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.3:50865)
0925 12:20:35.673516 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:35.673546 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":49,"threads_started":1}
I20260925 12:20:35.742854 17297 raft_consensus.cc:482] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:20:35.743010 17297 raft_consensus.cc:504] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260925 12:20:35.743312 17297 leader_election.cc:288] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c0e6524debb4ecf9571f9b526597f17 (127.0.98.2:39615), ef9694aa8ddb48819d8fbbacf1017fac (127.0.98.3:33243)
Sep 25 12:20:35 dist-test-slave-clmx krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 25 12:20:35 dist-test-slave-clmx krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260925 12:20:35.755698 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:35.743655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57991 (local address 127.0.98.3:33243)
0925 12:20:35.743820 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:35.743827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:35.743853 (+ 26us) server_negotiation.cc:407] Connection header received
0925 12:20:35.744371 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:35.744373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:35.744427 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:35.744572 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:35.745510 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:35.746807 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:35.748290 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:35.748897 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:35.749002 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:35.751794 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:35.751797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:35.751797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:35.751807 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:35.753608 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:35.754255 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:35.754257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:35.754257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:35.754367 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:35.754722 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:35.754723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:35.754723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:35.755031 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:35.755113 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:35.755445 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 12:20:35.755568 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260925 12:20:35.755698 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:35.743572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49501 (local address 127.0.98.2:39615)
0925 12:20:35.743764 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:35.743775 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:35.743824 (+ 49us) server_negotiation.cc:407] Connection header received
0925 12:20:35.744371 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:35.744373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:35.744427 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:35.744572 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:35.745510 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:35.746807 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:35.748290 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:35.748897 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:35.749002 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:35.751351 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:35.751355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:35.751355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:35.751365 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:35.753608 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:35.754247 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:35.754250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:35.754251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:35.754367 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:35.754712 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:35.754713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:35.754713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:35.755031 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:35.755113 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:35.755444 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 12:20:35.755568 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260925 12:20:35.755695 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:35.743424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39615 (local address 127.0.98.1:49501)
0925 12:20:35.743456 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:35.743468 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:35.743522 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:35.744637 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:35.744640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:35.745119 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:35.745386 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:35.745394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:35.746914 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:35.746915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:35.748201 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:35.748204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:35.749003 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:35.749004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:35.749155 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:35.749733 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:20:35.749740 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:35.751250 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:35.753723 (+ 2473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:35.753731 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:35.753733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:35.754159 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:35.754460 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:35.754461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:35.754461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:35.754631 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:35.755111 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:35.755118 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:35.755268 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 12:20:35.755372 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 12:20:35.755544 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16,"mutex_wait_us":84}
I20260925 12:20:35.755751 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:35.743482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33243 (local address 127.0.98.1:57991)
0925 12:20:35.743604 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:35.743613 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:35.743688 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:35.744637 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:35.744640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:35.745118 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:35.745386 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:35.745394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:35.746916 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:35.746916 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:35.748201 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:35.748204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:35.749401 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:35.749403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:35.749456 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:35.749911 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:20:35.749927 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:35.751698 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:35.753723 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:35.753731 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:35.753733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:35.754159 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:35.754460 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:35.754461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:35.754461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:35.754631 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:35.755111 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:35.755118 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:35.755268 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 12:20:35.755372 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 12:20:35.755609 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":185,"thread_start_us":69,"threads_started":1}
W20260925 12:20:35.756491 16911 leader_election.cc:341] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c0e6524debb4ecf9571f9b526597f17 (127.0.98.2:39615): Not found: Tablet not found: 80814af23d614ef9aa42426d1f0797fe
W20260925 12:20:35.756949 16913 leader_election.cc:341] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef9694aa8ddb48819d8fbbacf1017fac (127.0.98.3:33243): Not found: Tablet not found: 80814af23d614ef9aa42426d1f0797fe
I20260925 12:20:35.757011 16913 leader_election.cc:302] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c97a4eabb5d4282aae54e4aa821dd35; no voters: 1c0e6524debb4ecf9571f9b526597f17, ef9694aa8ddb48819d8fbbacf1017fac
I20260925 12:20:35.757131 17297 raft_consensus.cc:2726] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:20:36.497042 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:36.496833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.1:42511)
0925 12:20:36.496969 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:36.497001 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260925 12:20:36.556381 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:36.556224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.2:36281)
0925 12:20:36.556321 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:36.556349 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":51,"threads_started":1}
I20260925 12:20:36.674129 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:36.673961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.3:51087)
0925 12:20:36.674066 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:36.674098 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":54,"threads_started":1}
I20260925 12:20:37.497890 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:37.497607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.1:57713)
0925 12:20:37.497815 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:37.497848 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":67,"threads_started":1}
I20260925 12:20:37.556907 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:37.556754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.2:55141)
0925 12:20:37.556857 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:37.556877 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260925 12:20:37.558782 17306 raft_consensus.cc:482] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:20:37.558851 17306 raft_consensus.cc:504] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260925 12:20:37.559005 17306 leader_election.cc:288] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c0e6524debb4ecf9571f9b526597f17 (127.0.98.2:39615), ef9694aa8ddb48819d8fbbacf1017fac (127.0.98.3:33243)
W20260925 12:20:37.559419 16913 leader_election.cc:341] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef9694aa8ddb48819d8fbbacf1017fac (127.0.98.3:33243): Not found: Tablet not found: 80814af23d614ef9aa42426d1f0797fe
W20260925 12:20:37.559479 16911 leader_election.cc:341] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c0e6524debb4ecf9571f9b526597f17 (127.0.98.2:39615): Not found: Tablet not found: 80814af23d614ef9aa42426d1f0797fe
I20260925 12:20:37.559504 16913 leader_election.cc:302] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c97a4eabb5d4282aae54e4aa821dd35; no voters: 1c0e6524debb4ecf9571f9b526597f17, ef9694aa8ddb48819d8fbbacf1017fac
I20260925 12:20:37.559564 17306 raft_consensus.cc:2726] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:20:37.674690 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:37.674511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.3:47799)
0925 12:20:37.674622 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:37.674655 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
I20260925 12:20:38.498594 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:38.498406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.1:41431)
0925 12:20:38.498515 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:38.498548 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260925 12:20:38.557446 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:38.557254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.2:56247)
0925 12:20:38.557373 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:38.557400 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":52,"threads_started":1}
I20260925 12:20:38.675261 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:38.675086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.3:60027)
0925 12:20:38.675191 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:38.675223 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":52,"threads_started":1}
I20260925 12:20:39.499253 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:39.499053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.1:58907)
0925 12:20:39.499178 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:39.499210 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260925 12:20:39.558200 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:39.558024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.2:32797)
0925 12:20:39.558135 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:39.558166 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260925 12:20:39.675863 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:39.675679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.3:53391)
0925 12:20:39.675804 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:39.675831 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260925 12:20:40.500013 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:40.499793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.1:60095)
0925 12:20:40.499936 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:40.499969 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260925 12:20:40.552543 17315 raft_consensus.cc:482] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:20:40.552623 17315 raft_consensus.cc:504] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260925 12:20:40.552780 17315 leader_election.cc:288] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c0e6524debb4ecf9571f9b526597f17 (127.0.98.2:39615), ef9694aa8ddb48819d8fbbacf1017fac (127.0.98.3:33243)
W20260925 12:20:40.553131 16913 leader_election.cc:341] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef9694aa8ddb48819d8fbbacf1017fac (127.0.98.3:33243): Not found: Tablet not found: 80814af23d614ef9aa42426d1f0797fe
W20260925 12:20:40.553207 16911 leader_election.cc:341] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c0e6524debb4ecf9571f9b526597f17 (127.0.98.2:39615): Not found: Tablet not found: 80814af23d614ef9aa42426d1f0797fe
I20260925 12:20:40.553239 16911 leader_election.cc:302] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c97a4eabb5d4282aae54e4aa821dd35; no voters: 1c0e6524debb4ecf9571f9b526597f17, ef9694aa8ddb48819d8fbbacf1017fac
I20260925 12:20:40.553318 17315 raft_consensus.cc:2726] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:20:40.558745 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:40.558567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.2:60243)
0925 12:20:40.558677 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:40.558705 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":76,"threads_started":1}
I20260925 12:20:40.676384 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:40.676195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.3:38727)
0925 12:20:40.676307 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:40.676344 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
I20260925 12:20:41.500769 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:41.500527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.1:54161)
0925 12:20:41.500655 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:41.500709 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260925 12:20:41.559289 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:41.559102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.2:34933)
0925 12:20:41.559220 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:41.559252 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260925 12:20:41.676964 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:41.676788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.3:54573)
0925 12:20:41.676898 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:41.676930 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":50,"threads_started":1}
I20260925 12:20:42.501369 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:42.501176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.1:48881)
0925 12:20:42.501291 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:42.501324 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":53,"threads_started":1}
I20260925 12:20:42.559950 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:42.559754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.2:41037)
0925 12:20:42.559896 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:42.559920 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260925 12:20:42.677608 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:42.677444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.3:38579)
0925 12:20:42.677546 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:42.677576 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":52,"threads_started":1}
I20260925 12:20:43.502210 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:43.501891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.1:60885)
0925 12:20:43.502133 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:43.502166 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":70,"threads_started":1}
I20260925 12:20:43.560473 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:43.560305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.2:37983)
0925 12:20:43.560409 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:43.560439 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":52,"threads_started":1}
I20260925 12:20:43.678246 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:43.678057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.3:42117)
0925 12:20:43.678166 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:43.678211 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":52,"threads_started":1}
I20260925 12:20:44.502938 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:44.502644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.1:34249)
0925 12:20:44.502857 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:44.502891 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":62,"threads_started":1}
I20260925 12:20:44.553012 17328 raft_consensus.cc:482] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:20:44.553114 17328 raft_consensus.cc:504] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e6524debb4ecf9571f9b526597f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39615 } } peers { permanent_uuid: "ef9694aa8ddb48819d8fbbacf1017fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } } peers { permanent_uuid: "2c97a4eabb5d4282aae54e4aa821dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260925 12:20:44.553314 17328 leader_election.cc:288] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c0e6524debb4ecf9571f9b526597f17 (127.0.98.2:39615), ef9694aa8ddb48819d8fbbacf1017fac (127.0.98.3:33243)
W20260925 12:20:44.553681 16913 leader_election.cc:341] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef9694aa8ddb48819d8fbbacf1017fac (127.0.98.3:33243): Not found: Tablet not found: 80814af23d614ef9aa42426d1f0797fe
W20260925 12:20:44.553762 16911 leader_election.cc:341] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c0e6524debb4ecf9571f9b526597f17 (127.0.98.2:39615): Not found: Tablet not found: 80814af23d614ef9aa42426d1f0797fe
I20260925 12:20:44.553845 16913 leader_election.cc:302] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c97a4eabb5d4282aae54e4aa821dd35; no voters: 1c0e6524debb4ecf9571f9b526597f17, ef9694aa8ddb48819d8fbbacf1017fac
I20260925 12:20:44.553941 17328 raft_consensus.cc:2726] T 80814af23d614ef9aa42426d1f0797fe P 2c97a4eabb5d4282aae54e4aa821dd35 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:20:44.561033 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:44.560843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.2:48289)
0925 12:20:44.560941 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:44.560987 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":63,"threads_started":1}
I20260925 12:20:44.679019 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:44.678718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.3:37563)
0925 12:20:44.678942 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:44.678975 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":69,"threads_started":1}
I20260925 12:20:45.503793 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:45.503565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.1:53251)
0925 12:20:45.503713 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:45.503747 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260925 12:20:45.561684 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:45.561470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.2:33503)
0925 12:20:45.561612 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:45.561647 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260925 12:20:45.679736 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:45.679433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.3:39879)
0925 12:20:45.679612 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:45.679667 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":103,"threads_started":1}
I20260925 12:20:46.504427 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:46.504222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.1:52057)
0925 12:20:46.504348 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:46.504382 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
I20260925 12:20:46.562240 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:46.562069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.2:35367)
0925 12:20:46.562166 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:46.562196 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":50,"threads_started":1}
I20260925 12:20:46.680418 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:46.680245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.3:55889)
0925 12:20:46.680355 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:46.680384 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260925 12:20:47.505074 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:47.504863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.1:41811)
0925 12:20:47.504995 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:47.505028 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260925 12:20:47.562991 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:47.562770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.2:52041)
0925 12:20:47.562919 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:47.562952 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260925 12:20:47.680939 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:47.680774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.3:49337)
0925 12:20:47.680875 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:47.680905 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
I20260925 12:20:48.505753 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:48.505539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.1:34629)
0925 12:20:48.505673 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:48.505707 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260925 12:20:48.563779 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:48.563553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.2:58185)
0925 12:20:48.563706 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:48.563742 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":74,"threads_started":1}
I20260925 12:20:48.681511 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:48.681315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.3:36561)
0925 12:20:48.681423 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:48.681453 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
I20260925 12:20:49.506880 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:49.506615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.1:46291)
0925 12:20:49.506772 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:49.506836 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20260925 12:20:49.564492 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:49.564301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.2:54971)
0925 12:20:49.564419 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:49.564458 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":55,"threads_started":1}
I20260925 12:20:49.682191 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:49.682004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38775 (local address 127.0.98.3:41621)
0925 12:20:49.682116 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:49.682156 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":51,"threads_started":1}
I20260925 12:20:49.686721 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 16860
I20260925 12:20:49.691267 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 17029
I20260925 12:20:49.695313 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 17161
2026-09-25T12:20:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58252 ms)
[----------] 30 tests from SecurityITest (224699 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:20:49 dist-test-slave-clmx krb5kdc[17348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:20:49 dist-test-slave-clmx krb5kdc[17348](info): set up 2 sockets
Sep 25 12:20:49 dist-test-slave-clmx krb5kdc[17348](info): commencing operation
krb5kdc: starting...
W20260925 12:20:50.782970 392 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:20:50 dist-test-slave-clmx krb5kdc[17348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:20:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:20:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:20:50.888445 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34743
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:20:50.964485 17361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:50.964716 17361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:50.964762 17361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:50.965878 17361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:20:50.965957 17361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:50.965981 17361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:50.966009 17361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:20:50.966039 17361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:20:50.967443 17361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:50.967737 17361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:50.967955 17361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:50.972424 17366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:50.972434 17367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:50.972476 17369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:50.972494 17361 server_base.cc:1034] running on GCE node
I20260925 12:20:50.972883 17361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:50.973146 17361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:50.974314 17361 hybrid_clock.cc:648] HybridClock initialized: now 1790338850974304 us; error 31 us; skew 500 ppm
Sep 25 12:20:50 dist-test-slave-clmx krb5kdc[17348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:50.976598 17361 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:20:50.977057 17361 webserver.cc:460] Webserver started at http://127.0.98.62:37853/ using document root <none> and password file <none>
I20260925 12:20:50.977325 17361 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:50.977370 17361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:50.977528 17361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:50.978464 17361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "abcd9b3766854b5c9fa017243b85c237"
format_stamp: "Formatted at 2026-09-25 12:20:50 on dist-test-slave-clmx"
server_key: "2cd49448b6fb6e822a667a9d79bd2daa"
server_key_iv: "660563b619f38f0807cd13b37e2514d4"
server_key_version: "encryptionkey@0"
I20260925 12:20:50.978790 17361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "abcd9b3766854b5c9fa017243b85c237"
format_stamp: "Formatted at 2026-09-25 12:20:50 on dist-test-slave-clmx"
server_key: "2cd49448b6fb6e822a667a9d79bd2daa"
server_key_iv: "660563b619f38f0807cd13b37e2514d4"
server_key_version: "encryptionkey@0"
I20260925 12:20:50.980134 17361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:20:50.980806 17376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:50.980973 17361 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:20:50.981036 17361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "abcd9b3766854b5c9fa017243b85c237"
format_stamp: "Formatted at 2026-09-25 12:20:50 on dist-test-slave-clmx"
server_key: "2cd49448b6fb6e822a667a9d79bd2daa"
server_key_iv: "660563b619f38f0807cd13b37e2514d4"
server_key_version: "encryptionkey@0"
I20260925 12:20:50.981119 17361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:50.996563 17361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:50.998215 17361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:50.998366 17361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:51.002326 17361 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34743
I20260925 12:20:51.002399 17428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34743 every 8 connection(s)
I20260925 12:20:51.002676 17361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:20:51.003451 17429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:51.005540 17429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237: Bootstrap starting.
I20260925 12:20:51.006083 17429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:51.006358 17429 log.cc:824] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:51.006992 17429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237: No bootstrap required, opened a new log
I20260925 12:20:51.008595 17429 raft_consensus.cc:348] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcd9b3766854b5c9fa017243b85c237" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34743 } }
I20260925 12:20:51.008764 17429 raft_consensus.cc:374] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:51.008817 17429 raft_consensus.cc:729] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abcd9b3766854b5c9fa017243b85c237, State: Initialized, Role: FOLLOWER
I20260925 12:20:51.008977 17429 consensus_queue.cc:260] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcd9b3766854b5c9fa017243b85c237" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34743 } }
I20260925 12:20:51.009057 17429 raft_consensus.cc:388] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:20:51.009094 17429 raft_consensus.cc:482] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:20:51.009145 17429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:51.009625 17429 raft_consensus.cc:504] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcd9b3766854b5c9fa017243b85c237" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34743 } }
I20260925 12:20:51.009737 17429 leader_election.cc:302] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abcd9b3766854b5c9fa017243b85c237; no voters:
I20260925 12:20:51.009909 17429 leader_election.cc:288] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:20:51.009927 17432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:20:51.010092 17432 raft_consensus.cc:686] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237 [term 1 LEADER]: Becoming Leader. State: Replica: abcd9b3766854b5c9fa017243b85c237, State: Running, Role: LEADER
I20260925 12:20:51.010144 17429 sys_catalog.cc:564] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:20:51.010226 17432 consensus_queue.cc:237] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcd9b3766854b5c9fa017243b85c237" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34743 } }
I20260925 12:20:51.010602 17434 sys_catalog.cc:455] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abcd9b3766854b5c9fa017243b85c237. Latest consensus state: current_term: 1 leader_uuid: "abcd9b3766854b5c9fa017243b85c237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcd9b3766854b5c9fa017243b85c237" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34743 } } }
I20260925 12:20:51.010690 17434 sys_catalog.cc:458] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237 [sys.catalog]: This master's current role is: LEADER
I20260925 12:20:51.010743 17433 sys_catalog.cc:455] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abcd9b3766854b5c9fa017243b85c237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcd9b3766854b5c9fa017243b85c237" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34743 } } }
I20260925 12:20:51.010820 17433 sys_catalog.cc:458] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237 [sys.catalog]: This master's current role is: LEADER
I20260925 12:20:51.011090 17440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:20:51.011700 17440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:20:51.012703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 17361
I20260925 12:20:51.012816 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:20:51.013000 392 external_mini_cluster.cc:1428] Setting key 06febe629cd144a8004c50b753970780
I20260925 12:20:51.020105 17440 catalog_manager.cc:1347] Generated new cluster ID: a6db65534e4748279022f4a77ca86f74
I20260925 12:20:51.020203 17440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:20:51 dist-test-slave-clmx krb5kdc[17348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:20:51.027659 17440 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:20:51.028285 17440 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:20:51.030539 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.013461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48078 (local address 127.0.98.62:34743)
0925 12:20:51.019803 (+ 6342us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:51.019810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:51.019837 (+ 27us) server_negotiation.cc:407] Connection header received
0925 12:20:51.019928 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:51.019933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:51.019997 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:51.020142 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:51.020911 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.022257 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.023355 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.023966 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.024062 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.026839 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:51.026843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:51.026843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:51.026854 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:51.028937 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:51.029327 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:51.029330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:51.029330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:51.029451 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:51.029653 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:51.029654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:51.029654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:51.029931 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:51.030081 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:51.030171 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 12:20:51.030313 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6284,"thread_start_us":64,"threads_started":1}
I20260925 12:20:51.034533 17440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237: Generated new TSK 0
I20260925 12:20:51.034703 17440 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:20:51.062069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34743
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:20:51.140834 17455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:51.141021 17455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:51.141042 17455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:51.142118 17455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:51.142153 17455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:51.142179 17455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:20:51.143534 17455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:51.143764 17455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:51.143953 17455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:51.148506 17461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:51.148551 17460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:51.148521 17455 server_base.cc:1034] running on GCE node
W20260925 12:20:51.148506 17463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:51.148927 17455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:51.149121 17455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:51.150261 17455 hybrid_clock.cc:648] HybridClock initialized: now 1790338851150232 us; error 36 us; skew 500 ppm
Sep 25 12:20:51 dist-test-slave-clmx krb5kdc[17348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:51.152729 17455 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:20:51.153138 17455 webserver.cc:460] Webserver started at http://127.0.98.1:33681/ using document root <none> and password file <none>
I20260925 12:20:51.153318 17455 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:51.153357 17455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:51.153436 17455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:51.154345 17455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "a5b8e597ab4f4effbf91c199941141f5"
format_stamp: "Formatted at 2026-09-25 12:20:51 on dist-test-slave-clmx"
server_key: "6a5bffbae96c2b3b3f61d2a4407e3d01"
server_key_iv: "794ae7d789985cfd783ae3c9792c6f64"
server_key_version: "encryptionkey@0"
I20260925 12:20:51.154709 17455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a5b8e597ab4f4effbf91c199941141f5"
format_stamp: "Formatted at 2026-09-25 12:20:51 on dist-test-slave-clmx"
server_key: "6a5bffbae96c2b3b3f61d2a4407e3d01"
server_key_iv: "794ae7d789985cfd783ae3c9792c6f64"
server_key_version: "encryptionkey@0"
I20260925 12:20:51.155969 17455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:20:51.156584 17470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:51.156735 17455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:20:51.156827 17455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "a5b8e597ab4f4effbf91c199941141f5"
format_stamp: "Formatted at 2026-09-25 12:20:51 on dist-test-slave-clmx"
server_key: "6a5bffbae96c2b3b3f61d2a4407e3d01"
server_key_iv: "794ae7d789985cfd783ae3c9792c6f64"
server_key_version: "encryptionkey@0"
I20260925 12:20:51.156903 17455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:51.177965 17455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:51.180173 17455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:51.180333 17455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:51.180536 17455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:51.180822 17455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:51.180877 17455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:51.180917 17455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:51.180946 17455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:51.186712 17455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38161
I20260925 12:20:51.186764 17583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38161 every 8 connection(s)
I20260925 12:20:51.187052 17455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 12:20:51 dist-test-slave-clmx krb5kdc[17348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:20:51.195292 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.187698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50569 (local address 127.0.98.62:34743)
0925 12:20:51.187762 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:51.187764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:51.187954 (+ 190us) server_negotiation.cc:407] Connection header received
0925 12:20:51.188824 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:51.188825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:51.188834 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:51.188887 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:51.189413 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.189907 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.190826 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.191001 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.191047 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.192895 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:51.192897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:51.192897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:51.192904 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:51.194016 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:51.194392 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:51.194393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:51.194394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:51.194453 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:51.194685 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:51.194686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:51.194686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:51.194800 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:51.194873 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:51.195121 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 12:20:51.195205 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 12:20:51.195454 17586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.187677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34743 (local address 127.0.98.1:50569)
0925 12:20:51.187832 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:51.187870 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:51.188740 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:51.188935 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:51.188939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:51.189165 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:51.189342 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:51.189346 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.189978 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.189979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:51.190755 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:51.190757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.191066 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.191066 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:51.191153 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.191540 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:20:51.191547 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:51.192797 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:51.194098 (+ 1301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:51.194105 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:51.194107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:51.194316 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:51.194518 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:51.194519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:51.194519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:51.194618 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:51.194865 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:51.194871 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:51.194997 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 12:20:51.195098 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 12:20:51.195231 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260925 12:20:51.195912 17584 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34743
I20260925 12:20:51.195997 17584 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:51.196117 17584 heartbeater.cc:511] Master 127.0.98.62:34743 requested a full tablet report, sending...
I20260925 12:20:51.196465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 17455
I20260925 12:20:51.196574 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:20:51.196599 17393 ts_manager.cc:194] Registered new tserver with Master: a5b8e597ab4f4effbf91c199941141f5 (127.0.98.1:38161)
I20260925 12:20:51.196753 392 external_mini_cluster.cc:1428] Setting key 4071d590c3460111154bf88e6a54172b
I20260925 12:20:51.197307 17393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50569
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:20:51.218448 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34743
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:20:51.294528 17589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:51.294703 17589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:51.294724 17589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:51.295926 17589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:51.295965 17589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:51.295990 17589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:20:51.297430 17589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:51.297652 17589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:51.297847 17589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:51.302471 17595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:51.302476 17597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:51.302485 17594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:51.302810 17589 server_base.cc:1034] running on GCE node
I20260925 12:20:51.303006 17589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:51.303174 17589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:51.304308 17589 hybrid_clock.cc:648] HybridClock initialized: now 1790338851304298 us; error 25 us; skew 500 ppm
Sep 25 12:20:51 dist-test-slave-clmx krb5kdc[17348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:51.306685 17589 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:20:51.307106 17589 webserver.cc:460] Webserver started at http://127.0.98.2:36983/ using document root <none> and password file <none>
I20260925 12:20:51.307341 17589 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:51.307384 17589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:51.307525 17589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:51.308517 17589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "b58694315bc04a5a830d01f87fd3730e"
format_stamp: "Formatted at 2026-09-25 12:20:51 on dist-test-slave-clmx"
server_key: "33ec94332b5a7cb98efbb0a7c5549b90"
server_key_iv: "1b4f45ffab9e6443d6adf9fe93c42fd0"
server_key_version: "encryptionkey@0"
I20260925 12:20:51.308841 17589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b58694315bc04a5a830d01f87fd3730e"
format_stamp: "Formatted at 2026-09-25 12:20:51 on dist-test-slave-clmx"
server_key: "33ec94332b5a7cb98efbb0a7c5549b90"
server_key_iv: "1b4f45ffab9e6443d6adf9fe93c42fd0"
server_key_version: "encryptionkey@0"
I20260925 12:20:51.310067 17589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:20:51.310809 17604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:51.310994 17589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:20:51.311087 17589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "b58694315bc04a5a830d01f87fd3730e"
format_stamp: "Formatted at 2026-09-25 12:20:51 on dist-test-slave-clmx"
server_key: "33ec94332b5a7cb98efbb0a7c5549b90"
server_key_iv: "1b4f45ffab9e6443d6adf9fe93c42fd0"
server_key_version: "encryptionkey@0"
I20260925 12:20:51.311169 17589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:51.330639 17589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:51.332384 17589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:51.332530 17589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:51.332728 17589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:51.333029 17589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:51.333083 17589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:51.333119 17589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:51.333144 17589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:51.338531 17589 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35087
I20260925 12:20:51.338579 17717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35087 every 8 connection(s)
I20260925 12:20:51.338847 17589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:20:51.342478 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 17589
I20260925 12:20:51.342553 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:20:51.342742 392 external_mini_cluster.cc:1428] Setting key 19c6be1901705693a4d19a8def7eb1ba
Sep 25 12:20:51 dist-test-slave-clmx krb5kdc[17348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:20:51.348256 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.339453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34743 (local address 127.0.98.2:50113)
0925 12:20:51.339633 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:51.339672 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:51.340532 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:51.340745 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:51.340748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:51.340961 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:51.341117 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:51.341120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.341748 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.341748 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:51.342549 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:51.342551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.343035 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.343036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:51.343169 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.343694 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:20:51.343715 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:51.345394 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:51.347093 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:51.347098 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:51.347099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:51.347282 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:51.347479 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:51.347480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:51.347480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:51.347564 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:51.347848 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:51.347852 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:51.347948 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 12:20:51.348021 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 12:20:51.348123 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260925 12:20:51.348270 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.339463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50113 (local address 127.0.98.62:34743)
0925 12:20:51.339552 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:51.339554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:51.339755 (+ 201us) server_negotiation.cc:407] Connection header received
0925 12:20:51.340613 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:51.340614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:51.340624 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:51.340700 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:51.341179 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.341675 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.342661 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.342922 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.342988 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.345496 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:51.345497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:51.345497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:51.345505 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:51.347015 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:51.347354 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:51.347355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:51.347355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:51.347414 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:51.347640 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:51.347641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:51.347641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:51.347783 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:51.347854 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:51.348069 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 12:20:51.348152 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 12:20:51.348734 17718 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34743
I20260925 12:20:51.348830 17718 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:51.348981 17718 heartbeater.cc:511] Master 127.0.98.62:34743 requested a full tablet report, sending...
I20260925 12:20:51.349323 17393 ts_manager.cc:194] Registered new tserver with Master: b58694315bc04a5a830d01f87fd3730e (127.0.98.2:35087)
I20260925 12:20:51.349884 17393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50113
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:20:51.366209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34743
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:20:51.442039 17723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:51.442202 17723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:51.442224 17723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:51.443252 17723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:51.443296 17723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:51.443321 17723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:20:51.445175 17723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:51.445408 17723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:51.445641 17723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:51.450582 17731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:51.450604 17728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:51.450654 17729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:51.450814 17723 server_base.cc:1034] running on GCE node
I20260925 12:20:51.451017 17723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:51.451242 17723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:51.452389 17723 hybrid_clock.cc:648] HybridClock initialized: now 1790338851452376 us; error 27 us; skew 500 ppm
Sep 25 12:20:51 dist-test-slave-clmx krb5kdc[17348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:51.454682 17723 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:20:51.455122 17723 webserver.cc:460] Webserver started at http://127.0.98.3:41951/ using document root <none> and password file <none>
I20260925 12:20:51.455307 17723 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:51.455343 17723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:51.455420 17723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:51.456319 17723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "7ebc21da434847549f5dcab9cd54bc7a"
format_stamp: "Formatted at 2026-09-25 12:20:51 on dist-test-slave-clmx"
server_key: "4e6b4c9509d6e21ea2bc36e8916ebfc5"
server_key_iv: "61de976e658eebc9b5edf6c4cb9b5f05"
server_key_version: "encryptionkey@0"
I20260925 12:20:51.456570 17723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7ebc21da434847549f5dcab9cd54bc7a"
format_stamp: "Formatted at 2026-09-25 12:20:51 on dist-test-slave-clmx"
server_key: "4e6b4c9509d6e21ea2bc36e8916ebfc5"
server_key_iv: "61de976e658eebc9b5edf6c4cb9b5f05"
server_key_version: "encryptionkey@0"
I20260925 12:20:51.457674 17723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:20:51.458278 17738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:51.458432 17723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:20:51.458518 17723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "7ebc21da434847549f5dcab9cd54bc7a"
format_stamp: "Formatted at 2026-09-25 12:20:51 on dist-test-slave-clmx"
server_key: "4e6b4c9509d6e21ea2bc36e8916ebfc5"
server_key_iv: "61de976e658eebc9b5edf6c4cb9b5f05"
server_key_version: "encryptionkey@0"
I20260925 12:20:51.458563 17723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:51.476651 17723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:51.478744 17723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:51.478852 17723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:51.478996 17723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:51.479264 17723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:51.479295 17723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:51.479337 17723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:51.479362 17723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:51.485100 17723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42287
I20260925 12:20:51.485152 17851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42287 every 8 connection(s)
I20260925 12:20:51.485419 17723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:20:51.490216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 17723
I20260925 12:20:51.490320 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:20:51.490548 392 external_mini_cluster.cc:1428] Setting key 644166bf23fcc83488961cc2bb4495ef
Sep 25 12:20:51 dist-test-slave-clmx krb5kdc[17348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:20:51.494246 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.486064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41409 (local address 127.0.98.62:34743)
0925 12:20:51.486148 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:51.486151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:51.486319 (+ 168us) server_negotiation.cc:407] Connection header received
0925 12:20:51.487212 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:51.487213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:51.487225 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:51.487274 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:51.487914 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.488399 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.489351 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.489571 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.489656 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.491849 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:51.491851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:51.491851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:51.491857 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:51.492940 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:51.493358 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:51.493359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:51.493359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:51.493419 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:51.493659 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:51.493660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:51.493660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:51.493769 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:51.493843 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:51.494090 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 12:20:51.494162 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 12:20:51.494413 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.486023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34743 (local address 127.0.98.3:41409)
0925 12:20:51.486199 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:51.486241 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:51.487099 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:51.487335 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:51.487338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:51.487578 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:51.487821 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:51.487826 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.488503 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.488504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:51.489261 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:51.489263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.489663 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.489664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:51.489788 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.490231 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:20:51.490240 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:51.491744 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:51.493027 (+ 1283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:51.493033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:51.493035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:51.493235 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:51.493482 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:51.493483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:51.493484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:51.493588 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:51.493835 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:51.493841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:51.493968 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 12:20:51.494069 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 12:20:51.494206 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260925 12:20:51.494853 17852 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34743
I20260925 12:20:51.494920 17852 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:51.495090 17852 heartbeater.cc:511] Master 127.0.98.62:34743 requested a full tablet report, sending...
I20260925 12:20:51.495450 17393 ts_manager.cc:194] Registered new tserver with Master: 7ebc21da434847549f5dcab9cd54bc7a (127.0.98.3:42287)
I20260925 12:20:51.495929 17393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41409
I20260925 12:20:51.502174 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:20:51.509243 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.503428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48088 (local address 127.0.98.62:34743)
0925 12:20:51.503508 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:51.503510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:51.503646 (+ 136us) server_negotiation.cc:407] Connection header received
0925 12:20:51.503710 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:51.503711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:51.503721 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:51.503776 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:51.504352 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.504803 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.505537 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.505730 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.505814 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.506953 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:51.506954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:51.506954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:51.506960 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:51.508104 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:51.508484 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:51.508485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:51.508485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:51.508541 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:51.508734 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:51.508735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:51.508735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:51.508866 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:51.508983 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:51.509084 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 12:20:51.509153 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 12:20:51.510823 17393 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:20:51.512481 17393 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:20:51.518394 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.514635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35087 (local address 127.0.0.1:34460)
0925 12:20:51.514950 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:51.514957 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:51.514998 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:51.515200 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:51.515201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:51.515203 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:51.515252 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:51.515254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.516174 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.516175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:51.517250 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:51.517253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.518104 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.518105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:51.518186 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.518199 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 12:20:51.518280 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 12:20:51.518332 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":40,"threads_started":1}
I20260925 12:20:51.518427 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.514735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34460 (local address 127.0.98.2:35087)
0925 12:20:51.514935 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:51.514939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:51.514993 (+ 54us) server_negotiation.cc:407] Connection header received
0925 12:20:51.515092 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:51.515094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:51.515107 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:51.515186 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:51.515326 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.516078 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.517351 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.518012 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.518100 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.518157 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:51.518313 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 12:20:51.518354 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":89,"threads_started":1}
I20260925 12:20:51.518805 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.514468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42287 (local address 127.0.0.1:39732)
0925 12:20:51.514841 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:51.514850 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:51.514899 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:51.515091 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:51.515094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:51.515096 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:51.515356 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:51.515358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.516622 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.516623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:51.517515 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:51.517517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.518408 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.518409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:51.518604 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.518609 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:20:51.518675 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 12:20:51.518715 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":26,"thread_start_us":46,"threads_started":1}
I20260925 12:20:51.518838 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.514556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39732 (local address 127.0.98.3:42287)
0925 12:20:51.514708 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:51.514713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:51.514900 (+ 187us) server_negotiation.cc:407] Connection header received
0925 12:20:51.514965 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:51.514967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:51.514981 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:51.515066 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:51.515440 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.516503 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.517611 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.518254 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.518322 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.518373 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:51.518710 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 12:20:51.518752 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
I20260925 12:20:51.519153 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.514468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38161 (local address 127.0.0.1:47604)
0925 12:20:51.514731 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:51.514794 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:51.514857 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:51.515300 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:51.515301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:51.515303 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:51.515344 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:51.515346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.516611 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.516612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:51.517722 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:51.517724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.518776 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.518777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:51.518845 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.518848 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:20:51.519086 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 12:20:51.519114 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":24,"thread_start_us":52,"threads_started":1}
I20260925 12:20:51.519076 17652 tablet_service.cc:1467] Processing CreateTablet for tablet 086852008a1b4326a6165b9dd3edb1f3 (DEFAULT_TABLE table=test-table [id=c47940f474074516bbcc5a4d3a00b8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:51.519399 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.514671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.0.98.1:38161)
0925 12:20:51.514834 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:51.514838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:51.514849 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:20:51.515120 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:51.515123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:51.515140 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:51.515209 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:51.515423 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.516503 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.517800 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.518637 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.518694 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.518745 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:51.519279 (+ 534us) server_negotiation.cc:299] Negotiation successful
0925 12:20:51.519323 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20260925 12:20:51.519409 17652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086852008a1b4326a6165b9dd3edb1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:51.520035 17518 tablet_service.cc:1467] Processing CreateTablet for tablet 086852008a1b4326a6165b9dd3edb1f3 (DEFAULT_TABLE table=test-table [id=c47940f474074516bbcc5a4d3a00b8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:51.520339 17518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086852008a1b4326a6165b9dd3edb1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:51.522938 17866 tablet_bootstrap.cc:492] T 086852008a1b4326a6165b9dd3edb1f3 P b58694315bc04a5a830d01f87fd3730e: Bootstrap starting.
I20260925 12:20:51.523166 17786 tablet_service.cc:1467] Processing CreateTablet for tablet 086852008a1b4326a6165b9dd3edb1f3 (DEFAULT_TABLE table=test-table [id=c47940f474074516bbcc5a4d3a00b8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:51.523455 17786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086852008a1b4326a6165b9dd3edb1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:51.523578 17866 tablet_bootstrap.cc:654] T 086852008a1b4326a6165b9dd3edb1f3 P b58694315bc04a5a830d01f87fd3730e: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:51.523674 17867 tablet_bootstrap.cc:492] T 086852008a1b4326a6165b9dd3edb1f3 P a5b8e597ab4f4effbf91c199941141f5: Bootstrap starting.
I20260925 12:20:51.524094 17866 log.cc:824] T 086852008a1b4326a6165b9dd3edb1f3 P b58694315bc04a5a830d01f87fd3730e: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:51.524416 17867 tablet_bootstrap.cc:654] T 086852008a1b4326a6165b9dd3edb1f3 P a5b8e597ab4f4effbf91c199941141f5: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:51.524767 17866 tablet_bootstrap.cc:492] T 086852008a1b4326a6165b9dd3edb1f3 P b58694315bc04a5a830d01f87fd3730e: No bootstrap required, opened a new log
I20260925 12:20:51.524844 17866 ts_tablet_manager.cc:1397] T 086852008a1b4326a6165b9dd3edb1f3 P b58694315bc04a5a830d01f87fd3730e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 12:20:51.524966 17867 log.cc:824] T 086852008a1b4326a6165b9dd3edb1f3 P a5b8e597ab4f4effbf91c199941141f5: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:51.525681 17867 tablet_bootstrap.cc:492] T 086852008a1b4326a6165b9dd3edb1f3 P a5b8e597ab4f4effbf91c199941141f5: No bootstrap required, opened a new log
I20260925 12:20:51.525746 17867 ts_tablet_manager.cc:1397] T 086852008a1b4326a6165b9dd3edb1f3 P a5b8e597ab4f4effbf91c199941141f5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 12:20:51.526512 17866 raft_consensus.cc:348] T 086852008a1b4326a6165b9dd3edb1f3 P b58694315bc04a5a830d01f87fd3730e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b8e597ab4f4effbf91c199941141f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } } peers { permanent_uuid: "b58694315bc04a5a830d01f87fd3730e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35087 } } peers { permanent_uuid: "7ebc21da434847549f5dcab9cd54bc7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42287 } }
I20260925 12:20:51.526666 17866 raft_consensus.cc:374] T 086852008a1b4326a6165b9dd3edb1f3 P b58694315bc04a5a830d01f87fd3730e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:51.526710 17866 raft_consensus.cc:729] T 086852008a1b4326a6165b9dd3edb1f3 P b58694315bc04a5a830d01f87fd3730e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b58694315bc04a5a830d01f87fd3730e, State: Initialized, Role: FOLLOWER
I20260925 12:20:51.526819 17866 consensus_queue.cc:260] T 086852008a1b4326a6165b9dd3edb1f3 P b58694315bc04a5a830d01f87fd3730e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b8e597ab4f4effbf91c199941141f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } } peers { permanent_uuid: "b58694315bc04a5a830d01f87fd3730e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35087 } } peers { permanent_uuid: "7ebc21da434847549f5dcab9cd54bc7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42287 } }
I20260925 12:20:51.527104 17866 ts_tablet_manager.cc:1428] T 086852008a1b4326a6165b9dd3edb1f3 P b58694315bc04a5a830d01f87fd3730e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 12:20:51.527176 17870 tablet_bootstrap.cc:492] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a: Bootstrap starting.
I20260925 12:20:51.527235 17718 heartbeater.cc:503] Master 127.0.98.62:34743 was elected leader, sending a full tablet report...
I20260925 12:20:51.527797 17870 tablet_bootstrap.cc:654] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:51.527878 17867 raft_consensus.cc:348] T 086852008a1b4326a6165b9dd3edb1f3 P a5b8e597ab4f4effbf91c199941141f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b8e597ab4f4effbf91c199941141f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } } peers { permanent_uuid: "b58694315bc04a5a830d01f87fd3730e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35087 } } peers { permanent_uuid: "7ebc21da434847549f5dcab9cd54bc7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42287 } }
I20260925 12:20:51.528057 17867 raft_consensus.cc:374] T 086852008a1b4326a6165b9dd3edb1f3 P a5b8e597ab4f4effbf91c199941141f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:51.528102 17867 raft_consensus.cc:729] T 086852008a1b4326a6165b9dd3edb1f3 P a5b8e597ab4f4effbf91c199941141f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b8e597ab4f4effbf91c199941141f5, State: Initialized, Role: FOLLOWER
I20260925 12:20:51.528219 17867 consensus_queue.cc:260] T 086852008a1b4326a6165b9dd3edb1f3 P a5b8e597ab4f4effbf91c199941141f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b8e597ab4f4effbf91c199941141f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } } peers { permanent_uuid: "b58694315bc04a5a830d01f87fd3730e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35087 } } peers { permanent_uuid: "7ebc21da434847549f5dcab9cd54bc7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42287 } }
I20260925 12:20:51.528306 17870 log.cc:824] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:51.528501 17867 ts_tablet_manager.cc:1428] T 086852008a1b4326a6165b9dd3edb1f3 P a5b8e597ab4f4effbf91c199941141f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 12:20:51.528514 17584 heartbeater.cc:503] Master 127.0.98.62:34743 was elected leader, sending a full tablet report...
I20260925 12:20:51.528930 17870 tablet_bootstrap.cc:492] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a: No bootstrap required, opened a new log
I20260925 12:20:51.529009 17870 ts_tablet_manager.cc:1397] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:20:51.530318 17870 raft_consensus.cc:348] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b8e597ab4f4effbf91c199941141f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } } peers { permanent_uuid: "b58694315bc04a5a830d01f87fd3730e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35087 } } peers { permanent_uuid: "7ebc21da434847549f5dcab9cd54bc7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42287 } }
I20260925 12:20:51.530486 17870 raft_consensus.cc:374] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:51.530531 17870 raft_consensus.cc:729] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ebc21da434847549f5dcab9cd54bc7a, State: Initialized, Role: FOLLOWER
I20260925 12:20:51.530648 17870 consensus_queue.cc:260] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b8e597ab4f4effbf91c199941141f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } } peers { permanent_uuid: "b58694315bc04a5a830d01f87fd3730e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35087 } } peers { permanent_uuid: "7ebc21da434847549f5dcab9cd54bc7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42287 } }
I20260925 12:20:51.530894 17870 ts_tablet_manager.cc:1428] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:20:51.530926 17852 heartbeater.cc:503] Master 127.0.98.62:34743 was elected leader, sending a full tablet report...
W20260925 12:20:51.589313 17719 tablet.cc:2369] T 086852008a1b4326a6165b9dd3edb1f3 P b58694315bc04a5a830d01f87fd3730e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:20:51.613104 17874 raft_consensus.cc:482] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:20:51.613198 17874 raft_consensus.cc:504] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b8e597ab4f4effbf91c199941141f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } } peers { permanent_uuid: "b58694315bc04a5a830d01f87fd3730e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35087 } } peers { permanent_uuid: "7ebc21da434847549f5dcab9cd54bc7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42287 } }
I20260925 12:20:51.613509 17874 leader_election.cc:288] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5b8e597ab4f4effbf91c199941141f5 (127.0.98.1:38161), b58694315bc04a5a830d01f87fd3730e (127.0.98.2:35087)
I20260925 12:20:51.617324 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.613949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35911 (local address 127.0.98.2:35087)
0925 12:20:51.613989 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:51.613991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:51.614023 (+ 32us) server_negotiation.cc:407] Connection header received
0925 12:20:51.614090 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:51.614091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:51.614102 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:51.614171 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:51.614319 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.615008 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.616427 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.616972 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.617047 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.617056 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:51.617216 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 12:20:51.617254 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:20:51.617321 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.613762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42299 (local address 127.0.98.1:38161)
0925 12:20:51.613826 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:51.613830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:51.613844 (+ 14us) server_negotiation.cc:407] Connection header received
0925 12:20:51.613882 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:51.613883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:51.613896 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:51.613978 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:51.614160 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.614924 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.616427 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.616972 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.617047 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.617056 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:51.617216 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 12:20:51.617254 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 12:20:51.617324 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.613682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38161 (local address 127.0.98.3:42299)
0925 12:20:51.613744 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:51.613756 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:51.613798 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:51.614010 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:51.614011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:51.614012 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:51.614084 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:51.614086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.615018 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.615018 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:51.616321 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:51.616323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.617056 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.617057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:51.617140 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.617143 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:20:51.617193 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 12:20:51.617245 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":5}
I20260925 12:20:51.617340 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.613817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35087 (local address 127.0.98.3:35911)
0925 12:20:51.613972 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:51.613981 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:51.614032 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:51.614186 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:51.614187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:51.614188 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:51.614247 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:51.614248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.615077 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.615078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:51.616321 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:51.616323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.617056 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.617057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:51.617131 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.617135 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:20:51.617193 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 12:20:51.617239 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":37,"thread_start_us":96,"threads_started":1}
I20260925 12:20:51.617719 17538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086852008a1b4326a6165b9dd3edb1f3" candidate_uuid: "7ebc21da434847549f5dcab9cd54bc7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b8e597ab4f4effbf91c199941141f5" is_pre_election: true
I20260925 12:20:51.617832 17538 raft_consensus.cc:2454] T 086852008a1b4326a6165b9dd3edb1f3 P a5b8e597ab4f4effbf91c199941141f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ebc21da434847549f5dcab9cd54bc7a in term 0.
I20260925 12:20:51.617964 17672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086852008a1b4326a6165b9dd3edb1f3" candidate_uuid: "7ebc21da434847549f5dcab9cd54bc7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b58694315bc04a5a830d01f87fd3730e" is_pre_election: true
I20260925 12:20:51.618052 17742 leader_election.cc:302] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ebc21da434847549f5dcab9cd54bc7a, a5b8e597ab4f4effbf91c199941141f5; no voters:
I20260925 12:20:51.618099 17672 raft_consensus.cc:2454] T 086852008a1b4326a6165b9dd3edb1f3 P b58694315bc04a5a830d01f87fd3730e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ebc21da434847549f5dcab9cd54bc7a in term 0.
I20260925 12:20:51.618187 17874 raft_consensus.cc:2781] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:20:51.618250 17874 raft_consensus.cc:482] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:20:51.618266 17874 raft_consensus.cc:3037] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:51.618902 17874 raft_consensus.cc:504] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b8e597ab4f4effbf91c199941141f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } } peers { permanent_uuid: "b58694315bc04a5a830d01f87fd3730e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35087 } } peers { permanent_uuid: "7ebc21da434847549f5dcab9cd54bc7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42287 } }
I20260925 12:20:51.619019 17874 leader_election.cc:288] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [CANDIDATE]: Term 1 election: Requested vote from peers a5b8e597ab4f4effbf91c199941141f5 (127.0.98.1:38161), b58694315bc04a5a830d01f87fd3730e (127.0.98.2:35087)
I20260925 12:20:51.619172 17538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086852008a1b4326a6165b9dd3edb1f3" candidate_uuid: "7ebc21da434847549f5dcab9cd54bc7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b8e597ab4f4effbf91c199941141f5"
I20260925 12:20:51.619181 17672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086852008a1b4326a6165b9dd3edb1f3" candidate_uuid: "7ebc21da434847549f5dcab9cd54bc7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b58694315bc04a5a830d01f87fd3730e"
I20260925 12:20:51.619248 17672 raft_consensus.cc:3037] T 086852008a1b4326a6165b9dd3edb1f3 P b58694315bc04a5a830d01f87fd3730e [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:51.619270 17538 raft_consensus.cc:3037] T 086852008a1b4326a6165b9dd3edb1f3 P a5b8e597ab4f4effbf91c199941141f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:51.619989 17672 raft_consensus.cc:2454] T 086852008a1b4326a6165b9dd3edb1f3 P b58694315bc04a5a830d01f87fd3730e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ebc21da434847549f5dcab9cd54bc7a in term 1.
I20260925 12:20:51.620103 17538 raft_consensus.cc:2454] T 086852008a1b4326a6165b9dd3edb1f3 P a5b8e597ab4f4effbf91c199941141f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ebc21da434847549f5dcab9cd54bc7a in term 1.
I20260925 12:20:51.620157 17742 leader_election.cc:302] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ebc21da434847549f5dcab9cd54bc7a, b58694315bc04a5a830d01f87fd3730e; no voters:
I20260925 12:20:51.620267 17874 raft_consensus.cc:2781] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:20:51.620414 17874 raft_consensus.cc:686] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [term 1 LEADER]: Becoming Leader. State: Replica: 7ebc21da434847549f5dcab9cd54bc7a, State: Running, Role: LEADER
I20260925 12:20:51.620513 17874 consensus_queue.cc:237] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b8e597ab4f4effbf91c199941141f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } } peers { permanent_uuid: "b58694315bc04a5a830d01f87fd3730e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35087 } } peers { permanent_uuid: "7ebc21da434847549f5dcab9cd54bc7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42287 } }
I20260925 12:20:51.621308 17393 catalog_manager.cc:5495] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ebc21da434847549f5dcab9cd54bc7a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7ebc21da434847549f5dcab9cd54bc7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b8e597ab4f4effbf91c199941141f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b58694315bc04a5a830d01f87fd3730e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ebc21da434847549f5dcab9cd54bc7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42287 } health_report { overall_health: HEALTHY } } }
I20260925 12:20:51.633335 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.631229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39740 (local address 127.0.98.3:42287)
0925 12:20:51.631288 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:51.631289 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:51.631297 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:20:51.631338 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:51.631339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:51.631352 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:51.631410 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:20:51.631514 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.632033 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.632738 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.632910 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.632956 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.633070 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:20:51.633150 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:20:51.633217 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:51.633259 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 12:20:51.633273 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 12:20:51.634745 17538 raft_consensus.cc:1261] T 086852008a1b4326a6165b9dd3edb1f3 P a5b8e597ab4f4effbf91c199941141f5 [term 1 FOLLOWER]: Refusing update from remote peer 7ebc21da434847549f5dcab9cd54bc7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:20:51.634773 17672 raft_consensus.cc:1261] T 086852008a1b4326a6165b9dd3edb1f3 P b58694315bc04a5a830d01f87fd3730e [term 1 FOLLOWER]: Refusing update from remote peer 7ebc21da434847549f5dcab9cd54bc7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:20:51.635095 17876 consensus_queue.cc:1035] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b8e597ab4f4effbf91c199941141f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:20:51.635193 17874 consensus_queue.cc:1035] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b58694315bc04a5a830d01f87fd3730e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:20:51.637678 17879 mvcc.cc:205] Tried to move back new op lower bound from 7333227936293777408 to 7333227936238317568. Current Snapshot: MvccSnapshot[applied={T|T < 7333227936293777408}]
I20260925 12:20:51.639816 17393 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:20:51.639909 17393 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:20:51.640568 17393 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abcd9b3766854b5c9fa017243b85c237: Sending DeleteTablet for 3 replicas of tablet 086852008a1b4326a6165b9dd3edb1f3
I20260925 12:20:51.640898 17786 tablet_service.cc:1514] Processing DeleteTablet for tablet 086852008a1b4326a6165b9dd3edb1f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:20:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39732
I20260925 12:20:51.640956 17518 tablet_service.cc:1514] Processing DeleteTablet for tablet 086852008a1b4326a6165b9dd3edb1f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:20:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47604
I20260925 12:20:51.640955 17652 tablet_service.cc:1514] Processing DeleteTablet for tablet 086852008a1b4326a6165b9dd3edb1f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:20:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34460
I20260925 12:20:51.641207 17888 tablet_replica.cc:330] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a: stopping tablet replica
I20260925 12:20:51.641309 17888 raft_consensus.cc:2229] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:20:51.641317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 17455
I20260925 12:20:51.643388 17888 raft_consensus.cc:2258] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:20:51.643821 17888 ts_tablet_manager.cc:1905] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:20:51.645789 17888 ts_tablet_manager.cc:1918] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:20:51.645866 17888 log.cc:1196] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/086852008a1b4326a6165b9dd3edb1f3
I20260925 12:20:51.646138 17888 ts_tablet_manager.cc:1939] T 086852008a1b4326a6165b9dd3edb1f3 P 7ebc21da434847549f5dcab9cd54bc7a: Deleting consensus metadata
I20260925 12:20:51.646507 17378 catalog_manager.cc:4841] TS 7ebc21da434847549f5dcab9cd54bc7a (127.0.98.3:42287): tablet 086852008a1b4326a6165b9dd3edb1f3 (table test-table [id=c47940f474074516bbcc5a4d3a00b8cc]) successfully deleted
W20260925 12:20:51.647030 17380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38161 (error 108)
I20260925 12:20:51.647224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 17589
I20260925 12:20:51.647476 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.647366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38161 (local address 127.0.0.1:47612)
0925 12:20:51.647406 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:51.647442 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 12:20:51.647569 17380 catalog_manager.cc:4584] TS a5b8e597ab4f4effbf91c199941141f5 (127.0.98.1:38161): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 086852008a1b4326a6165b9dd3edb1f3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38161: connect: Connection refused (error 111)
I20260925 12:20:51.652438 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.652329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35087 (local address 127.0.0.1:34466)
0925 12:20:51.652380 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:51.652406 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 12:20:51.652469 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 17723
I20260925 12:20:51.656381 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 17361
2026-09-25T12:20:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1871 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T12:20:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:20:51Z Disabled control of system clock
I20260925 12:20:51.682240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41547
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 12:20:51.756925 17895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:51.757112 17895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:51.757130 17895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:51.758136 17895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:20:51.758164 17895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:51.758176 17895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:51.758188 17895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:20:51.758198 17895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:20:51.759410 17895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41547
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:51.759598 17895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:51.759888 17895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:51.764207 17900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:51.764233 17895 server_base.cc:1034] running on GCE node
W20260925 12:20:51.764237 17901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:51.764209 17903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:51.764613 17895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:51.764832 17895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:51.765959 17895 hybrid_clock.cc:648] HybridClock initialized: now 1790338851765949 us; error 26 us; skew 500 ppm
I20260925 12:20:51.767113 17895 webserver.cc:460] Webserver started at http://127.0.98.62:41297/ using document root <none> and password file <none>
I20260925 12:20:51.767339 17895 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:51.767400 17895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:51.767510 17895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:51.768558 17895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "efdfedf8d62349bdbd71f9e8ee5b7f6a"
format_stamp: "Formatted at 2026-09-25 12:20:51 on dist-test-slave-clmx"
server_key: "bf6d49e9b57e547626c38f59b8663ed6"
server_key_iv: "37487562a2812da4e6ab36f964c648ff"
server_key_version: "encryptionkey@0"
I20260925 12:20:51.768894 17895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "efdfedf8d62349bdbd71f9e8ee5b7f6a"
format_stamp: "Formatted at 2026-09-25 12:20:51 on dist-test-slave-clmx"
server_key: "bf6d49e9b57e547626c38f59b8663ed6"
server_key_iv: "37487562a2812da4e6ab36f964c648ff"
server_key_version: "encryptionkey@0"
I20260925 12:20:51.770102 17895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:20:51.770805 17909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:51.770956 17895 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:20:51.771049 17895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "efdfedf8d62349bdbd71f9e8ee5b7f6a"
format_stamp: "Formatted at 2026-09-25 12:20:51 on dist-test-slave-clmx"
server_key: "bf6d49e9b57e547626c38f59b8663ed6"
server_key_iv: "37487562a2812da4e6ab36f964c648ff"
server_key_version: "encryptionkey@0"
I20260925 12:20:51.771122 17895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:51.799983 17895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:51.800264 17895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:51.800412 17895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:51.804119 17895 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41547
I20260925 12:20:51.804154 17961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41547 every 8 connection(s)
I20260925 12:20:51.804471 17895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:20:51.805346 17962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:51.806118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 17895
I20260925 12:20:51.806221 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:20:51.806417 392 external_mini_cluster.cc:1428] Setting key 954763c39f547e5c0ce9a573924c14fc
I20260925 12:20:51.808482 17962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a: Bootstrap starting.
I20260925 12:20:51.809087 17962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:51.809350 17962 log.cc:824] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:51.810000 17962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a: No bootstrap required, opened a new log
I20260925 12:20:51.812112 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.806971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.0.98.62:41547)
0925 12:20:51.807162 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:51.807170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:51.807198 (+ 28us) server_negotiation.cc:407] Connection header received
0925 12:20:51.807672 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:51.807686 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:51.807762 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:51.807897 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:51.808078 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.809304 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.810078 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.810664 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.810808 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.811406 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:51.811411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:51.811412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:20:51.811602 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:51.811664 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:20:51.811689 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:51.811763 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:51.811817 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 12:20:51.811900 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260925 12:20:51.812161 17962 raft_consensus.cc:348] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efdfedf8d62349bdbd71f9e8ee5b7f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41547 } }
I20260925 12:20:51.812325 17962 raft_consensus.cc:374] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:51.812367 17962 raft_consensus.cc:729] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdfedf8d62349bdbd71f9e8ee5b7f6a, State: Initialized, Role: FOLLOWER
I20260925 12:20:51.812497 17962 consensus_queue.cc:260] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efdfedf8d62349bdbd71f9e8ee5b7f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41547 } }
I20260925 12:20:51.812577 17962 raft_consensus.cc:388] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:20:51.812614 17962 raft_consensus.cc:482] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:20:51.812659 17962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:51.813109 17962 raft_consensus.cc:504] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efdfedf8d62349bdbd71f9e8ee5b7f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41547 } }
I20260925 12:20:51.813227 17962 leader_election.cc:302] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efdfedf8d62349bdbd71f9e8ee5b7f6a; no voters:
I20260925 12:20:51.813405 17962 leader_election.cc:288] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:20:51.813444 17967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:20:51.813592 17967 raft_consensus.cc:686] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a [term 1 LEADER]: Becoming Leader. State: Replica: efdfedf8d62349bdbd71f9e8ee5b7f6a, State: Running, Role: LEADER
I20260925 12:20:51.813652 17962 sys_catalog.cc:564] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:20:51.813676 17967 consensus_queue.cc:237] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efdfedf8d62349bdbd71f9e8ee5b7f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41547 } }
I20260925 12:20:51.813992 17968 sys_catalog.cc:455] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efdfedf8d62349bdbd71f9e8ee5b7f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efdfedf8d62349bdbd71f9e8ee5b7f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41547 } } }
I20260925 12:20:51.814002 17969 sys_catalog.cc:455] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader efdfedf8d62349bdbd71f9e8ee5b7f6a. Latest consensus state: current_term: 1 leader_uuid: "efdfedf8d62349bdbd71f9e8ee5b7f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efdfedf8d62349bdbd71f9e8ee5b7f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41547 } } }
I20260925 12:20:51.814145 17968 sys_catalog.cc:458] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a [sys.catalog]: This master's current role is: LEADER
I20260925 12:20:51.814163 17969 sys_catalog.cc:458] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a [sys.catalog]: This master's current role is: LEADER
I20260925 12:20:51.814486 17976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:20:51.814922 17976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:20:51.816288 17976 catalog_manager.cc:1347] Generated new cluster ID: 2987dc71b4bd4b969ad5816c55084794
I20260925 12:20:51.816345 17976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:20:51.830318 17976 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:20:51.830845 17976 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:20:51.839777 17976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a: Generated new TSK 0
I20260925 12:20:51.839947 17976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:20:51.844172 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41547
--builtin_ntp_servers=127.0.98.20:36405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 12:20:51.923049 17986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:51.923257 17986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:51.923290 17986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:51.924474 17986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:51.924533 17986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:51.924574 17986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:20:51.925868 17986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:51.926111 17986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:51.926388 17986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:51.931188 17994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:51.931165 17991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:51.931180 17992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:51.931506 17986 server_base.cc:1034] running on GCE node
I20260925 12:20:51.931699 17986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:51.931973 17986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:51.933127 17986 hybrid_clock.cc:648] HybridClock initialized: now 1790338851933117 us; error 26 us; skew 500 ppm
I20260925 12:20:51.934319 17986 webserver.cc:460] Webserver started at http://127.0.98.1:37737/ using document root <none> and password file <none>
I20260925 12:20:51.934569 17986 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:51.934613 17986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:51.934748 17986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:51.935804 17986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "6646d04d184840e4a4b3f237ffe288ab"
format_stamp: "Formatted at 2026-09-25 12:20:51 on dist-test-slave-clmx"
server_key: "79fc4f8ba68acf8fbc7b0fa5960b3c96"
server_key_iv: "1497e0c07b19c7bfb158bae4e0a688f9"
server_key_version: "encryptionkey@0"
I20260925 12:20:51.936128 17986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6646d04d184840e4a4b3f237ffe288ab"
format_stamp: "Formatted at 2026-09-25 12:20:51 on dist-test-slave-clmx"
server_key: "79fc4f8ba68acf8fbc7b0fa5960b3c96"
server_key_iv: "1497e0c07b19c7bfb158bae4e0a688f9"
server_key_version: "encryptionkey@0"
I20260925 12:20:51.937316 17986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:20:51.937950 18000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:51.938134 17986 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:51.938212 17986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "6646d04d184840e4a4b3f237ffe288ab"
format_stamp: "Formatted at 2026-09-25 12:20:51 on dist-test-slave-clmx"
server_key: "79fc4f8ba68acf8fbc7b0fa5960b3c96"
server_key_iv: "1497e0c07b19c7bfb158bae4e0a688f9"
server_key_version: "encryptionkey@0"
I20260925 12:20:51.938272 17986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:51.958069 17986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:51.958313 17986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:51.958410 17986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:51.958564 17986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:51.958796 17986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:51.958823 17986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:51.958843 17986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:51.958855 17986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:51.964083 17986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44085
I20260925 12:20:51.964170 18113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44085 every 8 connection(s)
I20260925 12:20:51.964483 17986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:20:51.969671 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.965173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41447 (local address 127.0.98.62:41547)
0925 12:20:51.965213 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:51.965215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:51.965432 (+ 217us) server_negotiation.cc:407] Connection header received
0925 12:20:51.966066 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:51.966067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:51.966084 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:51.966134 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:51.966786 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.967280 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.968349 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.968541 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.968615 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.969352 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:51.969353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:51.969354 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:20:51.969359 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:51.969372 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:20:51.969376 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:51.969431 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:51.969568 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 12:20:51.969595 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:20:51.969861 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:51.965146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41547 (local address 127.0.98.1:41447)
0925 12:20:51.965320 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:51.965352 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:51.965910 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:51.966194 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:51.966198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:51.966207 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:51.966668 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:51.966672 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.967369 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.967371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:51.968240 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:51.968242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:51.968621 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:51.968622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:51.968747 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:51.969188 (+ 441us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:20:51.969189 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:51.969213 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:20:51.969233 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:51.969445 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:51.969450 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:51.969452 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:20:51.969557 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 12:20:51.969649 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260925 12:20:51.970328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 17986
I20260925 12:20:51.970424 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:20:51.970427 18114 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41547
I20260925 12:20:51.970517 18114 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:51.970649 392 external_mini_cluster.cc:1428] Setting key 53d665a18ca0e5a59651258fbc2116bc
I20260925 12:20:51.970707 18114 heartbeater.cc:511] Master 127.0.98.62:41547 requested a full tablet report, sending...
I20260925 12:20:51.971220 17925 ts_manager.cc:194] Registered new tserver with Master: 6646d04d184840e4a4b3f237ffe288ab (127.0.98.1:44085)
I20260925 12:20:51.971586 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41547
--builtin_ntp_servers=127.0.98.20:36405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 12:20:51.972091 17925 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41447
W20260925 12:20:52.049319 18117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:52.049544 18117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:52.049580 18117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:52.050693 18117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:52.050750 18117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:52.050798 18117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:20:52.052151 18117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:52.052397 18117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:52.052661 18117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:52.057262 18125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:52.057281 18122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:52.057269 18123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:52.057428 18117 server_base.cc:1034] running on GCE node
I20260925 12:20:52.057608 18117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:52.057879 18117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:52.059039 18117 hybrid_clock.cc:648] HybridClock initialized: now 1790338852059026 us; error 44 us; skew 500 ppm
I20260925 12:20:52.060329 18117 webserver.cc:460] Webserver started at http://127.0.98.2:35483/ using document root <none> and password file <none>
I20260925 12:20:52.060575 18117 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:52.060645 18117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:52.060791 18117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:52.061746 18117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "a6ab4cb8fbd24b1a8aeebe3761f13b99"
format_stamp: "Formatted at 2026-09-25 12:20:52 on dist-test-slave-clmx"
server_key: "4927a6a27f020db36aa2624b7228bea9"
server_key_iv: "7eda36914691e55c36eaa7530780c033"
server_key_version: "encryptionkey@0"
I20260925 12:20:52.062088 18117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a6ab4cb8fbd24b1a8aeebe3761f13b99"
format_stamp: "Formatted at 2026-09-25 12:20:52 on dist-test-slave-clmx"
server_key: "4927a6a27f020db36aa2624b7228bea9"
server_key_iv: "7eda36914691e55c36eaa7530780c033"
server_key_version: "encryptionkey@0"
I20260925 12:20:52.063294 18117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:20:52.064061 18131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:52.064221 18117 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:52.064320 18117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "a6ab4cb8fbd24b1a8aeebe3761f13b99"
format_stamp: "Formatted at 2026-09-25 12:20:52 on dist-test-slave-clmx"
server_key: "4927a6a27f020db36aa2624b7228bea9"
server_key_iv: "7eda36914691e55c36eaa7530780c033"
server_key_version: "encryptionkey@0"
I20260925 12:20:52.064399 18117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:52.092654 18117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:52.092978 18117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:52.093135 18117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:52.093374 18117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:52.093690 18117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:52.093743 18117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:52.093786 18117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:52.093814 18117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:52.099387 18117 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44247
I20260925 12:20:52.099438 18244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44247 every 8 connection(s)
I20260925 12:20:52.099779 18117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:20:52.104583 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:52.100500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45059 (local address 127.0.98.62:41547)
0925 12:20:52.100543 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:52.100545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:52.100782 (+ 237us) server_negotiation.cc:407] Connection header received
0925 12:20:52.101235 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:52.101236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:52.101246 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:52.101305 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:52.101899 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.102385 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.103291 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.103466 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.103513 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:52.104380 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:52.104381 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:52.104382 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:20:52.104385 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:52.104407 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:20:52.104411 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:52.104448 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:52.104510 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 12:20:52.104528 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:20:52.104774 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:52.100461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41547 (local address 127.0.98.2:45059)
0925 12:20:52.100643 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:52.100682 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:52.101155 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:52.101354 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:52.101358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:52.101365 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:52.101832 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:52.101836 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.102461 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.102462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:52.103219 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:52.103220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.103533 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.103534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:52.103695 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:52.104257 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:20:52.104257 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:52.104285 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:20:52.104306 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:52.104449 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:52.104451 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:52.104453 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:20:52.104498 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 12:20:52.104568 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260925 12:20:52.105233 18245 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41547
I20260925 12:20:52.105357 18245 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:52.105525 18245 heartbeater.cc:511] Master 127.0.98.62:41547 requested a full tablet report, sending...
I20260925 12:20:52.105880 17925 ts_manager.cc:194] Registered new tserver with Master: a6ab4cb8fbd24b1a8aeebe3761f13b99 (127.0.98.2:44247)
I20260925 12:20:52.106242 17925 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:45059
I20260925 12:20:52.106640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 18117
I20260925 12:20:52.106745 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:20:52.106946 392 external_mini_cluster.cc:1428] Setting key 630d8c88552827994088486158029483
I20260925 12:20:52.107882 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41547
--builtin_ntp_servers=127.0.98.20:36405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 12:20:52.184176 18248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:52.184347 18248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:52.184367 18248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:52.185411 18248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:52.185442 18248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:52.185467 18248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:20:52.186676 18248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:52.186852 18248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:52.187062 18248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:52.191565 18253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:52.191658 18254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:52.191584 18256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:52.191834 18248 server_base.cc:1034] running on GCE node
I20260925 12:20:52.192052 18248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:52.192298 18248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:52.193456 18248 hybrid_clock.cc:648] HybridClock initialized: now 1790338852193448 us; error 34 us; skew 500 ppm
I20260925 12:20:52.194579 18248 webserver.cc:460] Webserver started at http://127.0.98.3:39827/ using document root <none> and password file <none>
I20260925 12:20:52.194818 18248 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:52.194896 18248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:52.195024 18248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:52.196058 18248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "a04fd1541e2c40d3b6eb249b851db1cf"
format_stamp: "Formatted at 2026-09-25 12:20:52 on dist-test-slave-clmx"
server_key: "44a6a10b01cecc8d6ff5ce1365a4b450"
server_key_iv: "56f50d8a8c573ffb5a9ad90bef6dcd5f"
server_key_version: "encryptionkey@0"
I20260925 12:20:52.196395 18248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a04fd1541e2c40d3b6eb249b851db1cf"
format_stamp: "Formatted at 2026-09-25 12:20:52 on dist-test-slave-clmx"
server_key: "44a6a10b01cecc8d6ff5ce1365a4b450"
server_key_iv: "56f50d8a8c573ffb5a9ad90bef6dcd5f"
server_key_version: "encryptionkey@0"
I20260925 12:20:52.197570 18248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:20:52.198208 18262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:52.198378 18248 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:20:52.198467 18248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "a04fd1541e2c40d3b6eb249b851db1cf"
format_stamp: "Formatted at 2026-09-25 12:20:52 on dist-test-slave-clmx"
server_key: "44a6a10b01cecc8d6ff5ce1365a4b450"
server_key_iv: "56f50d8a8c573ffb5a9ad90bef6dcd5f"
server_key_version: "encryptionkey@0"
I20260925 12:20:52.198547 18248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:52.207860 18248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:52.208092 18248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:52.208222 18248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:52.208415 18248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:52.208695 18248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:52.208747 18248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:52.208788 18248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:52.208815 18248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:52.214260 18248 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37275
I20260925 12:20:52.214329 18375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37275 every 8 connection(s)
I20260925 12:20:52.214665 18248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:20:52.219154 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:52.215296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43767 (local address 127.0.98.62:41547)
0925 12:20:52.215377 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:52.215379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:52.215566 (+ 187us) server_negotiation.cc:407] Connection header received
0925 12:20:52.216039 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:52.216040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:52.216050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:52.216100 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:52.216675 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.217168 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.218057 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.218241 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.218291 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:52.218941 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:52.218942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:52.218942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:20:52.218946 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:52.218958 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:20:52.218962 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:52.219008 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:52.219067 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 12:20:52.219099 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 12:20:52.219313 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:52.215272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41547 (local address 127.0.98.3:43767)
0925 12:20:52.215458 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:52.215486 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:52.215949 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:52.216145 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:52.216149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:52.216157 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:52.216613 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:52.216617 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.217249 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.217250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:52.217988 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:52.217989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.218317 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.218317 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:52.218417 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:52.218823 (+ 406us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:20:52.218823 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:52.218847 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:20:52.218866 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:52.219010 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:52.219012 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:52.219014 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:20:52.219058 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 12:20:52.219136 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":108,"threads_started":1}
I20260925 12:20:52.219749 18376 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41547
I20260925 12:20:52.219827 18376 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:52.219952 18376 heartbeater.cc:511] Master 127.0.98.62:41547 requested a full tablet report, sending...
I20260925 12:20:52.220256 17925 ts_manager.cc:194] Registered new tserver with Master: a04fd1541e2c40d3b6eb249b851db1cf (127.0.98.3:37275)
I20260925 12:20:52.220620 17925 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:43767
I20260925 12:20:52.222062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 18248
I20260925 12:20:52.222137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:20:52.222345 392 external_mini_cluster.cc:1428] Setting key 6e8c8b212be4e6a745dfe4394f8e9e7a
I20260925 12:20:52.223027 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:20:52.227101 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:52.224372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57152 (local address 127.0.98.62:41547)
0925 12:20:52.224449 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:52.224451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:52.224545 (+ 94us) server_negotiation.cc:407] Connection header received
0925 12:20:52.224616 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:52.224617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:52.224627 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:52.224683 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:52.224853 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.225311 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.226032 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.226215 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.226286 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:52.226877 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:52.226878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:52.226879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:20:52.226883 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:52.226895 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:20:52.226898 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:52.226939 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:52.227007 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 12:20:52.227026 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 12:20:52.228649 17925 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:20:52.230067 17925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:20:52.235998 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:52.232044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44085 (local address 127.0.0.1:44488)
0925 12:20:52.232191 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:52.232232 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:52.232296 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:52.232802 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:52.232803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:52.232805 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:52.232840 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:52.232841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.234029 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.234030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:52.234892 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:52.234895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.235684 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.235685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:52.235761 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:52.235775 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 12:20:52.235860 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 12:20:52.235923 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":1,"thread_start_us":53,"threads_started":1}
I20260925 12:20:52.236016 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:52.232326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44488 (local address 127.0.98.1:44085)
0925 12:20:52.232623 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:52.232628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:52.232640 (+ 12us) server_negotiation.cc:407] Connection header received
0925 12:20:52.232690 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:52.232692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:52.232709 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:52.232796 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:52.232921 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.233969 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.234953 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.235569 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.235671 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:52.235755 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:52.235880 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 12:20:52.235915 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":85,"threads_started":1}
I20260925 12:20:52.236141 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:52.232258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44247 (local address 127.0.0.1:39026)
0925 12:20:52.232410 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:52.232421 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:52.232459 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:52.232980 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:52.232981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:52.232983 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:52.233031 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:52.233032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.233874 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.233875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:52.235143 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:52.235145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.235928 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.235929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:52.235981 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:52.235985 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:20:52.236017 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 12:20:52.236055 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":94,"thread_start_us":64,"threads_started":1}
I20260925 12:20:52.236232 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:52.232044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37275 (local address 127.0.0.1:37654)
0925 12:20:52.232196 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:52.232233 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:52.232295 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:52.232673 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:52.232676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:52.232677 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:52.232738 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:52.232740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.233992 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.233993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:52.235128 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:52.235131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.236036 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.236037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:52.236102 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:52.236106 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:20:52.236138 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 12:20:52.236174 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":6,"thread_start_us":85,"threads_started":1}
I20260925 12:20:52.236467 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:52.232136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37654 (local address 127.0.98.3:37275)
0925 12:20:52.232525 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:52.232530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:52.232540 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:20:52.232575 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:52.232576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:52.232590 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:52.232667 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:52.232842 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.233901 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.235230 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.235940 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.236284 (+ 344us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:52.236344 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:52.236363 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 12:20:52.236387 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":89,"threads_started":1}
I20260925 12:20:52.237087 18310 tablet_service.cc:1467] Processing CreateTablet for tablet a7713f5df0304f128fdaa44b13e600ae (DEFAULT_TABLE table=test-table [id=2a6dddc452a64a3881ddaefb6f42dcc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:52.237118 18048 tablet_service.cc:1467] Processing CreateTablet for tablet a7713f5df0304f128fdaa44b13e600ae (DEFAULT_TABLE table=test-table [id=2a6dddc452a64a3881ddaefb6f42dcc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:52.237437 18048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7713f5df0304f128fdaa44b13e600ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:52.237439 18310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7713f5df0304f128fdaa44b13e600ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:52.239717 18390 tablet_bootstrap.cc:492] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf: Bootstrap starting.
I20260925 12:20:52.239981 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:52.232460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.0.98.2:44247)
0925 12:20:52.232635 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:52.232639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:52.232649 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:20:52.232690 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:52.232692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:52.232876 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:52.233103 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:52.233147 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.233784 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.235245 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.235832 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.236206 (+ 374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:52.236300 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:52.236328 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 12:20:52.236369 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260925 12:20:52.240391 18391 tablet_bootstrap.cc:492] T a7713f5df0304f128fdaa44b13e600ae P 6646d04d184840e4a4b3f237ffe288ab: Bootstrap starting.
I20260925 12:20:52.240494 18390 tablet_bootstrap.cc:654] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:52.240577 18179 tablet_service.cc:1467] Processing CreateTablet for tablet a7713f5df0304f128fdaa44b13e600ae (DEFAULT_TABLE table=test-table [id=2a6dddc452a64a3881ddaefb6f42dcc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:52.240852 18390 log.cc:824] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:52.240895 18179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7713f5df0304f128fdaa44b13e600ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:52.241099 18391 tablet_bootstrap.cc:654] T a7713f5df0304f128fdaa44b13e600ae P 6646d04d184840e4a4b3f237ffe288ab: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:52.241462 18391 log.cc:824] T a7713f5df0304f128fdaa44b13e600ae P 6646d04d184840e4a4b3f237ffe288ab: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:52.241688 18390 tablet_bootstrap.cc:492] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf: No bootstrap required, opened a new log
I20260925 12:20:52.241792 18390 ts_tablet_manager.cc:1397] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:20:52.242199 18391 tablet_bootstrap.cc:492] T a7713f5df0304f128fdaa44b13e600ae P 6646d04d184840e4a4b3f237ffe288ab: No bootstrap required, opened a new log
I20260925 12:20:52.242268 18391 ts_tablet_manager.cc:1397] T a7713f5df0304f128fdaa44b13e600ae P 6646d04d184840e4a4b3f237ffe288ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:20:52.243759 18394 tablet_bootstrap.cc:492] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99: Bootstrap starting.
I20260925 12:20:52.244302 18394 tablet_bootstrap.cc:654] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:52.244568 18394 log.cc:824] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:52.244489 18390 raft_consensus.cc:348] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a04fd1541e2c40d3b6eb249b851db1cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37275 } } peers { permanent_uuid: "a6ab4cb8fbd24b1a8aeebe3761f13b99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44247 } } peers { permanent_uuid: "6646d04d184840e4a4b3f237ffe288ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260925 12:20:52.244643 18390 raft_consensus.cc:374] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:52.244690 18390 raft_consensus.cc:729] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04fd1541e2c40d3b6eb249b851db1cf, State: Initialized, Role: FOLLOWER
I20260925 12:20:52.244820 18390 consensus_queue.cc:260] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a04fd1541e2c40d3b6eb249b851db1cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37275 } } peers { permanent_uuid: "a6ab4cb8fbd24b1a8aeebe3761f13b99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44247 } } peers { permanent_uuid: "6646d04d184840e4a4b3f237ffe288ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260925 12:20:52.245071 18390 ts_tablet_manager.cc:1428] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260925 12:20:52.245002 18391 raft_consensus.cc:348] T a7713f5df0304f128fdaa44b13e600ae P 6646d04d184840e4a4b3f237ffe288ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a04fd1541e2c40d3b6eb249b851db1cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37275 } } peers { permanent_uuid: "a6ab4cb8fbd24b1a8aeebe3761f13b99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44247 } } peers { permanent_uuid: "6646d04d184840e4a4b3f237ffe288ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260925 12:20:52.245141 18391 raft_consensus.cc:374] T a7713f5df0304f128fdaa44b13e600ae P 6646d04d184840e4a4b3f237ffe288ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:52.245193 18391 raft_consensus.cc:729] T a7713f5df0304f128fdaa44b13e600ae P 6646d04d184840e4a4b3f237ffe288ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6646d04d184840e4a4b3f237ffe288ab, State: Initialized, Role: FOLLOWER
I20260925 12:20:52.245209 18376 heartbeater.cc:503] Master 127.0.98.62:41547 was elected leader, sending a full tablet report...
I20260925 12:20:52.245287 18394 tablet_bootstrap.cc:492] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99: No bootstrap required, opened a new log
I20260925 12:20:52.245352 18394 ts_tablet_manager.cc:1397] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:20:52.245342 18391 consensus_queue.cc:260] T a7713f5df0304f128fdaa44b13e600ae P 6646d04d184840e4a4b3f237ffe288ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a04fd1541e2c40d3b6eb249b851db1cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37275 } } peers { permanent_uuid: "a6ab4cb8fbd24b1a8aeebe3761f13b99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44247 } } peers { permanent_uuid: "6646d04d184840e4a4b3f237ffe288ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260925 12:20:52.245595 18391 ts_tablet_manager.cc:1428] T a7713f5df0304f128fdaa44b13e600ae P 6646d04d184840e4a4b3f237ffe288ab: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 12:20:52.245756 18114 heartbeater.cc:503] Master 127.0.98.62:41547 was elected leader, sending a full tablet report...
I20260925 12:20:52.247334 18394 raft_consensus.cc:348] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a04fd1541e2c40d3b6eb249b851db1cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37275 } } peers { permanent_uuid: "a6ab4cb8fbd24b1a8aeebe3761f13b99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44247 } } peers { permanent_uuid: "6646d04d184840e4a4b3f237ffe288ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260925 12:20:52.247519 18394 raft_consensus.cc:374] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:52.247566 18394 raft_consensus.cc:729] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6ab4cb8fbd24b1a8aeebe3761f13b99, State: Initialized, Role: FOLLOWER
I20260925 12:20:52.247689 18394 consensus_queue.cc:260] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a04fd1541e2c40d3b6eb249b851db1cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37275 } } peers { permanent_uuid: "a6ab4cb8fbd24b1a8aeebe3761f13b99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44247 } } peers { permanent_uuid: "6646d04d184840e4a4b3f237ffe288ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260925 12:20:52.247895 18394 ts_tablet_manager.cc:1428] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:20:52.248025 18245 heartbeater.cc:503] Master 127.0.98.62:41547 was elected leader, sending a full tablet report...
W20260925 12:20:52.350292 18246 tablet.cc:2369] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 12:20:52.465157 18115 tablet.cc:2369] T a7713f5df0304f128fdaa44b13e600ae P 6646d04d184840e4a4b3f237ffe288ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 12:20:52.465232 18377 tablet.cc:2369] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:20:52.592833 18398 raft_consensus.cc:482] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:20:52.592940 18398 raft_consensus.cc:504] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a04fd1541e2c40d3b6eb249b851db1cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37275 } } peers { permanent_uuid: "a6ab4cb8fbd24b1a8aeebe3761f13b99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44247 } } peers { permanent_uuid: "6646d04d184840e4a4b3f237ffe288ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260925 12:20:52.593245 18398 leader_election.cc:288] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04fd1541e2c40d3b6eb249b851db1cf (127.0.98.3:37275), 6646d04d184840e4a4b3f237ffe288ab (127.0.98.1:44085)
I20260925 12:20:52.596940 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:52.593489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38449 (local address 127.0.98.1:44085)
0925 12:20:52.593551 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:52.593553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:52.593647 (+ 94us) server_negotiation.cc:407] Connection header received
0925 12:20:52.593683 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:52.593684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:52.593693 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:52.593748 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:52.593907 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.594627 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.596082 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.596605 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.596680 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:52.596688 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:52.596837 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 12:20:52.596874 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 12:20:52.596940 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:52.593489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53607 (local address 127.0.98.3:37275)
0925 12:20:52.593544 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:52.593547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:52.593590 (+ 43us) server_negotiation.cc:407] Connection header received
0925 12:20:52.593636 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:52.593637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:52.593647 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:52.593703 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:52.593911 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.594625 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.596082 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.596605 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.596680 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:52.596688 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:52.596840 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 12:20:52.596874 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:20:52.596951 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:52.593414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37275 (local address 127.0.98.2:53607)
0925 12:20:52.593476 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:52.593487 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:52.593541 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:52.593742 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:52.593743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:52.593745 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:52.593837 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:52.593838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.594703 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.594703 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:52.595995 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:52.595998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.596690 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.596691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:52.596761 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:52.596763 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:20:52.596829 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 12:20:52.596859 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":70}
I20260925 12:20:52.596956 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:52.593414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44085 (local address 127.0.98.2:38449)
0925 12:20:52.593584 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:52.593592 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:52.593636 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:52.593757 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:52.593757 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:52.593758 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:52.593831 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:52.593833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.594703 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.594703 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:52.595995 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:52.595998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.596700 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.596701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:52.596759 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:52.596763 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:20:52.596818 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 12:20:52.596858 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":113,"thread_start_us":64,"threads_started":1}
I20260925 12:20:52.597313 18330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7713f5df0304f128fdaa44b13e600ae" candidate_uuid: "a6ab4cb8fbd24b1a8aeebe3761f13b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04fd1541e2c40d3b6eb249b851db1cf" is_pre_election: true
I20260925 12:20:52.597499 18330 raft_consensus.cc:2454] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6ab4cb8fbd24b1a8aeebe3761f13b99 in term 0.
I20260925 12:20:52.597527 18068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7713f5df0304f128fdaa44b13e600ae" candidate_uuid: "a6ab4cb8fbd24b1a8aeebe3761f13b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6646d04d184840e4a4b3f237ffe288ab" is_pre_election: true
I20260925 12:20:52.597662 18068 raft_consensus.cc:2454] T a7713f5df0304f128fdaa44b13e600ae P 6646d04d184840e4a4b3f237ffe288ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6ab4cb8fbd24b1a8aeebe3761f13b99 in term 0.
I20260925 12:20:52.597705 18133 leader_election.cc:302] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04fd1541e2c40d3b6eb249b851db1cf, a6ab4cb8fbd24b1a8aeebe3761f13b99; no voters:
I20260925 12:20:52.597810 18398 raft_consensus.cc:2781] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:20:52.597862 18398 raft_consensus.cc:482] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:20:52.597887 18398 raft_consensus.cc:3037] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:52.598560 18398 raft_consensus.cc:504] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a04fd1541e2c40d3b6eb249b851db1cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37275 } } peers { permanent_uuid: "a6ab4cb8fbd24b1a8aeebe3761f13b99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44247 } } peers { permanent_uuid: "6646d04d184840e4a4b3f237ffe288ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260925 12:20:52.598699 18398 leader_election.cc:288] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [CANDIDATE]: Term 1 election: Requested vote from peers a04fd1541e2c40d3b6eb249b851db1cf (127.0.98.3:37275), 6646d04d184840e4a4b3f237ffe288ab (127.0.98.1:44085)
I20260925 12:20:52.598838 18068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7713f5df0304f128fdaa44b13e600ae" candidate_uuid: "a6ab4cb8fbd24b1a8aeebe3761f13b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6646d04d184840e4a4b3f237ffe288ab"
I20260925 12:20:52.598855 18330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7713f5df0304f128fdaa44b13e600ae" candidate_uuid: "a6ab4cb8fbd24b1a8aeebe3761f13b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04fd1541e2c40d3b6eb249b851db1cf"
I20260925 12:20:52.598912 18330 raft_consensus.cc:3037] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:52.598913 18068 raft_consensus.cc:3037] T a7713f5df0304f128fdaa44b13e600ae P 6646d04d184840e4a4b3f237ffe288ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:52.599807 18068 raft_consensus.cc:2454] T a7713f5df0304f128fdaa44b13e600ae P 6646d04d184840e4a4b3f237ffe288ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6ab4cb8fbd24b1a8aeebe3761f13b99 in term 1.
I20260925 12:20:52.599815 18330 raft_consensus.cc:2454] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6ab4cb8fbd24b1a8aeebe3761f13b99 in term 1.
I20260925 12:20:52.599968 18133 leader_election.cc:302] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04fd1541e2c40d3b6eb249b851db1cf, a6ab4cb8fbd24b1a8aeebe3761f13b99; no voters:
I20260925 12:20:52.600070 18398 raft_consensus.cc:2781] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:20:52.600236 18398 raft_consensus.cc:686] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [term 1 LEADER]: Becoming Leader. State: Replica: a6ab4cb8fbd24b1a8aeebe3761f13b99, State: Running, Role: LEADER
I20260925 12:20:52.600346 18398 consensus_queue.cc:237] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a04fd1541e2c40d3b6eb249b851db1cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37275 } } peers { permanent_uuid: "a6ab4cb8fbd24b1a8aeebe3761f13b99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44247 } } peers { permanent_uuid: "6646d04d184840e4a4b3f237ffe288ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260925 12:20:52.601184 17926 catalog_manager.cc:5495] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6ab4cb8fbd24b1a8aeebe3761f13b99 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a6ab4cb8fbd24b1a8aeebe3761f13b99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a04fd1541e2c40d3b6eb249b851db1cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6ab4cb8fbd24b1a8aeebe3761f13b99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6646d04d184840e4a4b3f237ffe288ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } health_report { overall_health: UNKNOWN } } }
I20260925 12:20:52.673342 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:52.671253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.0.98.2:44247)
0925 12:20:52.671325 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:52.671327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:52.671350 (+ 23us) server_negotiation.cc:407] Connection header received
0925 12:20:52.671385 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:52.671385 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:52.671403 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:52.671455 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:20:52.671566 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.672029 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.672700 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:52.672876 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:52.672921 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:52.673040 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:20:52.673146 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:20:52.673200 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:52.673249 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 12:20:52.673263 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 12:20:52.675101 18330 raft_consensus.cc:1261] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf [term 1 FOLLOWER]: Refusing update from remote peer a6ab4cb8fbd24b1a8aeebe3761f13b99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:20:52.675138 18068 raft_consensus.cc:1261] T a7713f5df0304f128fdaa44b13e600ae P 6646d04d184840e4a4b3f237ffe288ab [term 1 FOLLOWER]: Refusing update from remote peer a6ab4cb8fbd24b1a8aeebe3761f13b99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:20:52.675474 18400 consensus_queue.cc:1035] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6646d04d184840e4a4b3f237ffe288ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:20:52.675587 18398 consensus_queue.cc:1035] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04fd1541e2c40d3b6eb249b851db1cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:20:52.677995 18403 mvcc.cc:205] Tried to move back new op lower bound from 7333227940554838016 to 7333227940251688960. Current Snapshot: MvccSnapshot[applied={T|T < 7333227940554838016}]
I20260925 12:20:52.678795 18404 mvcc.cc:205] Tried to move back new op lower bound from 7333227940554838016 to 7333227940251688960. Current Snapshot: MvccSnapshot[applied={T|T < 7333227940554838016}]
I20260925 12:20:52.678890 18405 mvcc.cc:205] Tried to move back new op lower bound from 7333227940554838016 to 7333227940251688960. Current Snapshot: MvccSnapshot[applied={T|T < 7333227940554838016}]
I20260925 12:20:52.680054 17925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:20:52.680151 17925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:20:52.680915 17925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P efdfedf8d62349bdbd71f9e8ee5b7f6a: Sending DeleteTablet for 3 replicas of tablet a7713f5df0304f128fdaa44b13e600ae
I20260925 12:20:52.681200 18310 tablet_service.cc:1514] Processing DeleteTablet for tablet a7713f5df0304f128fdaa44b13e600ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:20:52 UTC) from {username='slave'} at 127.0.0.1:37654
I20260925 12:20:52.681226 18048 tablet_service.cc:1514] Processing DeleteTablet for tablet a7713f5df0304f128fdaa44b13e600ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:20:52 UTC) from {username='slave'} at 127.0.0.1:44488
I20260925 12:20:52.681252 18179 tablet_service.cc:1514] Processing DeleteTablet for tablet a7713f5df0304f128fdaa44b13e600ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:20:52 UTC) from {username='slave'} at 127.0.0.1:39026
I20260925 12:20:52.681571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 17986
I20260925 12:20:52.681496 18413 tablet_replica.cc:330] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf: stopping tablet replica
I20260925 12:20:52.683059 18413 raft_consensus.cc:2229] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:20:52.683166 18413 raft_consensus.cc:2258] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:20:52.683312 18415 tablet_replica.cc:330] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99: stopping tablet replica
I20260925 12:20:52.683396 18415 raft_consensus.cc:2229] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:20:52.683480 18415 raft_consensus.cc:2258] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:20:52.683879 18413 ts_tablet_manager.cc:1905] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:20:52.683984 18415 ts_tablet_manager.cc:1905] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:20:52.685681 18413 ts_tablet_manager.cc:1918] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:20:52.685811 18413 log.cc:1196] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/a7713f5df0304f128fdaa44b13e600ae
I20260925 12:20:52.685858 18415 ts_tablet_manager.cc:1918] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:20:52.685928 18415 log.cc:1196] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/a7713f5df0304f128fdaa44b13e600ae
I20260925 12:20:52.686165 18415 ts_tablet_manager.cc:1939] T a7713f5df0304f128fdaa44b13e600ae P a6ab4cb8fbd24b1a8aeebe3761f13b99: Deleting consensus metadata
I20260925 12:20:52.686236 18413 ts_tablet_manager.cc:1939] T a7713f5df0304f128fdaa44b13e600ae P a04fd1541e2c40d3b6eb249b851db1cf: Deleting consensus metadata
I20260925 12:20:52.686633 17911 catalog_manager.cc:4841] TS a6ab4cb8fbd24b1a8aeebe3761f13b99 (127.0.98.2:44247): tablet a7713f5df0304f128fdaa44b13e600ae (table test-table [id=2a6dddc452a64a3881ddaefb6f42dcc3]) successfully deleted
I20260925 12:20:52.686720 17911 catalog_manager.cc:4841] TS a04fd1541e2c40d3b6eb249b851db1cf (127.0.98.3:37275): tablet a7713f5df0304f128fdaa44b13e600ae (table test-table [id=2a6dddc452a64a3881ddaefb6f42dcc3]) successfully deleted
W20260925 12:20:52.687482 17913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44085 (error 108)
I20260925 12:20:52.687737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 18117
I20260925 12:20:52.687927 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:52.687823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44085 (local address 127.0.0.1:42326)
0925 12:20:52.687859 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:52.687900 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260925 12:20:52.689785 17913 catalog_manager.cc:4584] TS 6646d04d184840e4a4b3f237ffe288ab (127.0.98.1:44085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a7713f5df0304f128fdaa44b13e600ae: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44085: connect: Connection refused (error 111)
I20260925 12:20:52.692943 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 18248
I20260925 12:20:52.696959 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 17895
2026-09-25T12:20:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1039 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:20:52 dist-test-slave-clmx krb5kdc[18419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:20:52 dist-test-slave-clmx krb5kdc[18419](info): set up 2 sockets
Sep 25 12:20:52 dist-test-slave-clmx krb5kdc[18419](info): commencing operation
krb5kdc: starting...
W20260925 12:20:53.699263 392 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:20:53 dist-test-slave-clmx krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:20:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:20:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:20:53.804286 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46165
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:20:53.880386 18432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:53.880599 18432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:53.880636 18432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:53.881712 18432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:20:53.881778 18432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:53.881810 18432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:53.881842 18432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:20:53.881863 18432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:20:53.883313 18432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:53.883554 18432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:53.883792 18432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:53.888286 18438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:53.888316 18437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:53.888290 18440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:53.888463 18432 server_base.cc:1034] running on GCE node
I20260925 12:20:53.888723 18432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:53.888979 18432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:53.890132 18432 hybrid_clock.cc:648] HybridClock initialized: now 1790338853890116 us; error 28 us; skew 500 ppm
Sep 25 12:20:53 dist-test-slave-clmx krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:53.892490 18432 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:20:53.892882 18432 webserver.cc:460] Webserver started at http://127.0.98.62:36127/ using document root <none> and password file <none>
I20260925 12:20:53.893105 18432 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:53.893172 18432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:53.893313 18432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:53.894279 18432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "463bdc7ebf6f42c2baecda5a288f7b2a"
format_stamp: "Formatted at 2026-09-25 12:20:53 on dist-test-slave-clmx"
server_key: "5a57ece7b24d4eb9d247cb4432ac3d16"
server_key_iv: "c3e662659283358cb8a6f3aace14481d"
server_key_version: "encryptionkey@0"
I20260925 12:20:53.894651 18432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "463bdc7ebf6f42c2baecda5a288f7b2a"
format_stamp: "Formatted at 2026-09-25 12:20:53 on dist-test-slave-clmx"
server_key: "5a57ece7b24d4eb9d247cb4432ac3d16"
server_key_iv: "c3e662659283358cb8a6f3aace14481d"
server_key_version: "encryptionkey@0"
I20260925 12:20:53.895927 18432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:20:53.896620 18447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:53.896775 18432 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:20:53.896883 18432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "463bdc7ebf6f42c2baecda5a288f7b2a"
format_stamp: "Formatted at 2026-09-25 12:20:53 on dist-test-slave-clmx"
server_key: "5a57ece7b24d4eb9d247cb4432ac3d16"
server_key_iv: "c3e662659283358cb8a6f3aace14481d"
server_key_version: "encryptionkey@0"
I20260925 12:20:53.896951 18432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:53.922595 18432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:53.924490 18432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:53.924649 18432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:53.928517 18432 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46165
I20260925 12:20:53.928555 18499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46165 every 8 connection(s)
I20260925 12:20:53.928933 18432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:20:53.929720 18500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:53.931903 18500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a: Bootstrap starting.
I20260925 12:20:53.932441 18500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:53.932720 18500 log.cc:824] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:53.933350 18500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a: No bootstrap required, opened a new log
I20260925 12:20:53.934821 18500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463bdc7ebf6f42c2baecda5a288f7b2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46165 } }
I20260925 12:20:53.934944 18500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:53.934966 18500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 463bdc7ebf6f42c2baecda5a288f7b2a, State: Initialized, Role: FOLLOWER
I20260925 12:20:53.935149 18500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463bdc7ebf6f42c2baecda5a288f7b2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46165 } }
I20260925 12:20:53.935228 18500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:20:53.935277 18500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:20:53.935324 18500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:53.935876 18500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463bdc7ebf6f42c2baecda5a288f7b2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46165 } }
I20260925 12:20:53.935981 18500 leader_election.cc:302] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 463bdc7ebf6f42c2baecda5a288f7b2a; no voters:
I20260925 12:20:53.936157 18500 leader_election.cc:288] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:20:53.936231 18503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:20:53.936388 18503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a [term 1 LEADER]: Becoming Leader. State: Replica: 463bdc7ebf6f42c2baecda5a288f7b2a, State: Running, Role: LEADER
I20260925 12:20:53.936416 18500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:20:53.936479 18503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463bdc7ebf6f42c2baecda5a288f7b2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46165 } }
I20260925 12:20:53.936842 18505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 463bdc7ebf6f42c2baecda5a288f7b2a. Latest consensus state: current_term: 1 leader_uuid: "463bdc7ebf6f42c2baecda5a288f7b2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463bdc7ebf6f42c2baecda5a288f7b2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46165 } } }
I20260925 12:20:53.936924 18505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a [sys.catalog]: This master's current role is: LEADER
I20260925 12:20:53.937023 18504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "463bdc7ebf6f42c2baecda5a288f7b2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463bdc7ebf6f42c2baecda5a288f7b2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46165 } } }
I20260925 12:20:53.937107 18504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a [sys.catalog]: This master's current role is: LEADER
I20260925 12:20:53.937134 18511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:20:53.937601 18511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:20:53.938469 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 18432
I20260925 12:20:53.938570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:20:53.938745 392 external_mini_cluster.cc:1428] Setting key 707dc6cd98676493f86de16e1886173c
I20260925 12:20:53.939287 18511 catalog_manager.cc:1347] Generated new cluster ID: 9cdb424f2511418799ddf604866104be
I20260925 12:20:53.939343 18511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:20:53 dist-test-slave-clmx krb5kdc[18419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:20:53.951418 18511 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:20:53.952010 18511 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:20:53.952399 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:53.939160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48374 (local address 127.0.98.62:46165)
0925 12:20:53.943796 (+ 4636us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:53.943802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:53.943824 (+ 22us) server_negotiation.cc:407] Connection header received
0925 12:20:53.943912 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:53.943917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:53.943984 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:53.944116 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:53.944766 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:53.945894 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:53.946626 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:53.947009 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:53.947119 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:53.949236 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:53.949240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:53.949241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:53.949257 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:53.950837 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:53.951230 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:53.951233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:53.951233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:53.951337 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:53.951561 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:53.951562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:53.951562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:53.951810 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:53.951945 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:53.952053 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 12:20:53.952186 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4604,"thread_start_us":73,"threads_started":1}
I20260925 12:20:53.963124 18511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a: Generated new TSK 0
I20260925 12:20:53.963260 18511 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:20:53.995124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46165
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:20:54.069761 18526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:54.069979 18526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:54.070017 18526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:54.071141 18526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:54.071203 18526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:54.071245 18526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:20:54.072687 18526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:54.072933 18526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:54.073141 18526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:54.077559 18531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:54.077508 18534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:54.077531 18532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:54.077682 18526 server_base.cc:1034] running on GCE node
I20260925 12:20:54.077888 18526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:54.078069 18526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:54.079198 18526 hybrid_clock.cc:648] HybridClock initialized: now 1790338854079187 us; error 26 us; skew 500 ppm
Sep 25 12:20:54 dist-test-slave-clmx krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:54.081403 18526 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:20:54.081759 18526 webserver.cc:460] Webserver started at http://127.0.98.1:35081/ using document root <none> and password file <none>
I20260925 12:20:54.081943 18526 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:54.081982 18526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:54.082108 18526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:54.082989 18526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "0dfc09dd231742a79967cc8b42321e75"
format_stamp: "Formatted at 2026-09-25 12:20:54 on dist-test-slave-clmx"
server_key: "0032b24b368b64a11b9fe6adce85222e"
server_key_iv: "e32101c7569f5b70bba892def3d71d73"
server_key_version: "encryptionkey@0"
I20260925 12:20:54.083371 18526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0dfc09dd231742a79967cc8b42321e75"
format_stamp: "Formatted at 2026-09-25 12:20:54 on dist-test-slave-clmx"
server_key: "0032b24b368b64a11b9fe6adce85222e"
server_key_iv: "e32101c7569f5b70bba892def3d71d73"
server_key_version: "encryptionkey@0"
I20260925 12:20:54.084630 18526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:20:54.085304 18541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:54.085458 18526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:20:54.085549 18526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "0dfc09dd231742a79967cc8b42321e75"
format_stamp: "Formatted at 2026-09-25 12:20:54 on dist-test-slave-clmx"
server_key: "0032b24b368b64a11b9fe6adce85222e"
server_key_iv: "e32101c7569f5b70bba892def3d71d73"
server_key_version: "encryptionkey@0"
I20260925 12:20:54.085620 18526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:54.115748 18526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:54.117372 18526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:54.117511 18526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:54.117695 18526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:54.117950 18526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:54.117996 18526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:54.118033 18526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:54.118058 18526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:54.123373 18526 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32875
I20260925 12:20:54.123409 18654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32875 every 8 connection(s)
I20260925 12:20:54.123723 18526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 12:20:54 dist-test-slave-clmx krb5kdc[18419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:20:54.129616 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 18526
I20260925 12:20:54.129956 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:20:54.130177 392 external_mini_cluster.cc:1428] Setting key 2a1898611ca14e8b31b5cc87e4af0804
I20260925 12:20:54.132226 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.124401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34303 (local address 127.0.98.62:46165)
0925 12:20:54.124464 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:54.124466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:54.124621 (+ 155us) server_negotiation.cc:407] Connection header received
0925 12:20:54.125476 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:54.125477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:54.125485 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:54.125536 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:54.126042 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.126507 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.127359 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.127537 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.127579 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:54.129497 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:54.129498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:54.129498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:54.129503 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:54.130979 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:54.131337 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:54.131338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:54.131339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:54.131390 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:54.131642 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:54.131642 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:54.131643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:54.131743 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:54.131810 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:54.132058 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 12:20:54.132144 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 12:20:54.132449 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.124354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46165 (local address 127.0.98.1:34303)
0925 12:20:54.124507 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:54.124543 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:54.125398 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:54.125584 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:54.125587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:54.125801 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:54.125965 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:54.125969 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.126580 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.126580 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:54.127293 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:54.127295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.127595 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.127596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:54.127734 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:54.128206 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:20:54.128213 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:54.129416 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:54.131058 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:54.131064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:54.131066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:54.131260 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:54.131449 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:54.131450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:54.131450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:54.131549 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:54.131813 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:54.131826 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:54.131942 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 12:20:54.132038 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 12:20:54.132167 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260925 12:20:54.132889 18655 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46165
I20260925 12:20:54.132970 18655 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:54.133121 18655 heartbeater.cc:511] Master 127.0.98.62:46165 requested a full tablet report, sending...
I20260925 12:20:54.133531 18464 ts_manager.cc:194] Registered new tserver with Master: 0dfc09dd231742a79967cc8b42321e75 (127.0.98.1:32875)
I20260925 12:20:54.134070 18464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34303
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:20:54.152230 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46165
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:20:54.226522 18660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:54.226693 18660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:54.226715 18660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:54.227790 18660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:54.227825 18660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:54.227849 18660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:20:54.229197 18660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:54.229420 18660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:54.229625 18660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:54.234248 18668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:54.234273 18666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:54.234280 18665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:54.234434 18660 server_base.cc:1034] running on GCE node
I20260925 12:20:54.234632 18660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:54.234848 18660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:54.235977 18660 hybrid_clock.cc:648] HybridClock initialized: now 1790338854235968 us; error 24 us; skew 500 ppm
Sep 25 12:20:54 dist-test-slave-clmx krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:54.238343 18660 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:20:54.238754 18660 webserver.cc:460] Webserver started at http://127.0.98.2:43443/ using document root <none> and password file <none>
I20260925 12:20:54.238952 18660 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:54.239027 18660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:54.239159 18660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:54.240139 18660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "f86a52e11072476699f18e221a390a15"
format_stamp: "Formatted at 2026-09-25 12:20:54 on dist-test-slave-clmx"
server_key: "ab805c1af08f3d45704a1368e3ffc7cb"
server_key_iv: "1ae5fd4d52eac1e65a686e21b202c849"
server_key_version: "encryptionkey@0"
I20260925 12:20:54.240484 18660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f86a52e11072476699f18e221a390a15"
format_stamp: "Formatted at 2026-09-25 12:20:54 on dist-test-slave-clmx"
server_key: "ab805c1af08f3d45704a1368e3ffc7cb"
server_key_iv: "1ae5fd4d52eac1e65a686e21b202c849"
server_key_version: "encryptionkey@0"
I20260925 12:20:54.241669 18660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:20:54.242257 18675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:54.242403 18660 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:20:54.242488 18660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "f86a52e11072476699f18e221a390a15"
format_stamp: "Formatted at 2026-09-25 12:20:54 on dist-test-slave-clmx"
server_key: "ab805c1af08f3d45704a1368e3ffc7cb"
server_key_iv: "1ae5fd4d52eac1e65a686e21b202c849"
server_key_version: "encryptionkey@0"
I20260925 12:20:54.242560 18660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:54.254117 18660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:54.256273 18660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:54.256444 18660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:54.256650 18660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:54.256919 18660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:54.256973 18660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:54.257010 18660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:54.257036 18660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:54.262490 18660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33607
I20260925 12:20:54.262537 18788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33607 every 8 connection(s)
I20260925 12:20:54.262794 18660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:20:54.265950 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 18660
I20260925 12:20:54.266034 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:20:54.266251 392 external_mini_cluster.cc:1428] Setting key 81aa7630daa5176f5a603942c9d5ede1
Sep 25 12:20:54 dist-test-slave-clmx krb5kdc[18419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:20:54.272013 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.263377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52315 (local address 127.0.98.62:46165)
0925 12:20:54.263437 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:54.263440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:54.263653 (+ 213us) server_negotiation.cc:407] Connection header received
0925 12:20:54.264551 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:54.264552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:54.264565 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:54.264616 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:54.265142 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.265618 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.266800 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.266994 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.267062 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:54.269230 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:54.269232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:54.269232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:54.269239 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:54.270613 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:54.271049 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:54.271050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:54.271051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:54.271106 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:54.271335 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:54.271336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:54.271336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:54.271436 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:54.271508 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:54.271852 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 12:20:54.271930 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 12:20:54.272184 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.263357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46165 (local address 127.0.98.2:52315)
0925 12:20:54.263529 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:54.263560 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:54.264466 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:54.264681 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:54.264684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:54.264914 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:54.265076 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:54.265080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.265693 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.265693 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:54.266709 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:54.266711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.267080 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.267081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:54.267201 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:54.267669 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:20:54.267678 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:54.269138 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:54.270716 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:54.270722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:54.270724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:54.270955 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:54.271169 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:54.271169 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:54.271170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:54.271270 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:54.271500 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:54.271505 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:54.271700 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 12:20:54.271827 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 12:20:54.271967 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":96,"threads_started":1}
I20260925 12:20:54.272683 18789 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46165
I20260925 12:20:54.272774 18789 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:54.272939 18789 heartbeater.cc:511] Master 127.0.98.62:46165 requested a full tablet report, sending...
I20260925 12:20:54.273295 18464 ts_manager.cc:194] Registered new tserver with Master: f86a52e11072476699f18e221a390a15 (127.0.98.2:33607)
I20260925 12:20:54.273787 18464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52315
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:20:54.289837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46165
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:20:54.365041 18794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:54.365232 18794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:54.365272 18794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:54.366307 18794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:54.366339 18794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:54.366364 18794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:20:54.367745 18794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:54.367897 18794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:54.368046 18794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:54.372483 18802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:54.372501 18800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:54.372506 18799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:54.372768 18794 server_base.cc:1034] running on GCE node
I20260925 12:20:54.372958 18794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:54.373121 18794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:54.374289 18794 hybrid_clock.cc:648] HybridClock initialized: now 1790338854374271 us; error 27 us; skew 500 ppm
Sep 25 12:20:54 dist-test-slave-clmx krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:54.376569 18794 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:20:54.376969 18794 webserver.cc:460] Webserver started at http://127.0.98.3:36775/ using document root <none> and password file <none>
I20260925 12:20:54.377175 18794 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:54.377221 18794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:54.377354 18794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:54.378262 18794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "51f81649ddd8452bbe4a03b4eed9b7b6"
format_stamp: "Formatted at 2026-09-25 12:20:54 on dist-test-slave-clmx"
server_key: "7ab294a665fdf0531cabf83ce3891e1f"
server_key_iv: "80af7faef4dadf6eb31bcb891d4abe49"
server_key_version: "encryptionkey@0"
I20260925 12:20:54.378577 18794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "51f81649ddd8452bbe4a03b4eed9b7b6"
format_stamp: "Formatted at 2026-09-25 12:20:54 on dist-test-slave-clmx"
server_key: "7ab294a665fdf0531cabf83ce3891e1f"
server_key_iv: "80af7faef4dadf6eb31bcb891d4abe49"
server_key_version: "encryptionkey@0"
I20260925 12:20:54.379798 18794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:20:54.380450 18809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:54.380632 18794 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:20:54.380698 18794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "51f81649ddd8452bbe4a03b4eed9b7b6"
format_stamp: "Formatted at 2026-09-25 12:20:54 on dist-test-slave-clmx"
server_key: "7ab294a665fdf0531cabf83ce3891e1f"
server_key_iv: "80af7faef4dadf6eb31bcb891d4abe49"
server_key_version: "encryptionkey@0"
I20260925 12:20:54.380781 18794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:54.404202 18794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:54.405786 18794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:54.405923 18794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:54.406116 18794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:54.406381 18794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:54.406412 18794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:54.406431 18794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:54.406474 18794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:54.412129 18794 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36247
I20260925 12:20:54.412201 18922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36247 every 8 connection(s)
I20260925 12:20:54.412470 18794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:20:54.413733 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 18794
I20260925 12:20:54.413846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:20:54.414047 392 external_mini_cluster.cc:1428] Setting key 5098be8c4fd7da793681d216c9a33435
Sep 25 12:20:54 dist-test-slave-clmx krb5kdc[18419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:20:54.421499 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.413275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43265 (local address 127.0.98.62:46165)
0925 12:20:54.413349 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:54.413351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:54.413539 (+ 188us) server_negotiation.cc:407] Connection header received
0925 12:20:54.414523 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:54.414524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:54.414536 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:54.414579 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:54.415135 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.415681 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.416623 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.416824 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.416912 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:54.418822 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:54.418824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:54.418824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:54.418832 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:54.420214 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:54.420609 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:54.420610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:54.420611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:54.420669 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:54.420913 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:54.420914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:54.420914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:54.421048 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:54.421116 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:54.421350 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 12:20:54.421418 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 12:20:54.421674 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.413249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46165 (local address 127.0.98.3:43265)
0925 12:20:54.413424 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:54.413454 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:54.414420 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:54.414646 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:54.414649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:54.414865 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:54.415038 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:54.415050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.415779 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.415781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:54.416532 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:54.416534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.416914 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.416915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:54.417028 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:54.417490 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:20:54.417498 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:54.418725 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:54.420320 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:54.420326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:54.420328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:54.420533 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:54.420735 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:54.420736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:54.420736 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:54.420846 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:54.421109 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:54.421114 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:54.421234 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 12:20:54.421329 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 12:20:54.421455 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":106,"threads_started":1}
I20260925 12:20:54.422103 18923 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46165
I20260925 12:20:54.422195 18923 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:54.422374 18923 heartbeater.cc:511] Master 127.0.98.62:46165 requested a full tablet report, sending...
I20260925 12:20:54.422719 18464 ts_manager.cc:194] Registered new tserver with Master: 51f81649ddd8452bbe4a03b4eed9b7b6 (127.0.98.3:36247)
I20260925 12:20:54.423038 18464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43265
I20260925 12:20:54.425560 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:20:54.432467 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.426738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48386 (local address 127.0.98.62:46165)
0925 12:20:54.426800 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:54.426802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:54.426894 (+ 92us) server_negotiation.cc:407] Connection header received
0925 12:20:54.426926 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:54.426926 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:54.426934 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:54.426969 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:54.427454 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.427945 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.428602 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.428769 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.428821 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:54.429960 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:54.429961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:54.429961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:54.429967 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:54.431238 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:54.431608 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:54.431609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:54.431610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:54.431761 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:54.431975 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:54.431975 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:54.431975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:54.432101 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:54.432215 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:54.432282 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 12:20:54.432336 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 12:20:54.433912 18464 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:20:54.435480 18464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:20:54.441283 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.437707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36247 (local address 127.0.0.1:37814)
0925 12:20:54.437969 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:54.437977 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:54.438021 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:54.438277 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:54.438280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:54.438282 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:54.438345 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:54.438346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.439303 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.439304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:54.440158 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:54.440160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.441031 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.441032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:54.441106 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:54.441133 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 12:20:54.441173 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 12:20:54.441216 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":3,"thread_start_us":48,"threads_started":1}
I20260925 12:20:54.441660 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.437888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.0.98.3:36247)
0925 12:20:54.438115 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:54.438119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:54.438128 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:20:54.438158 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:54.438160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:54.438174 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:54.438414 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:54.438485 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.439209 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.440251 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.440948 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.441479 (+ 531us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:54.441535 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:54.441557 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 12:20:54.441594 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":71,"threads_started":1}
I20260925 12:20:54.441665 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.437706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33607 (local address 127.0.0.1:46856)
0925 12:20:54.437801 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:54.437837 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:54.437905 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:54.438447 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:54.438447 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:54.438449 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:54.438483 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:54.438483 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.439426 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.439427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:54.440628 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:54.440630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.441491 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.441492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:54.441549 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:54.441551 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:20:54.441579 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 12:20:54.441612 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":43,"threads_started":1}
I20260925 12:20:54.441984 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.437842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.0.98.2:33607)
0925 12:20:54.438281 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:54.438285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:54.438297 (+ 12us) server_negotiation.cc:407] Connection header received
0925 12:20:54.438337 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:54.438339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:54.438353 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:54.438449 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:54.438525 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.439313 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.440720 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.441402 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.441808 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:54.441875 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:54.441897 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 12:20:54.441935 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":99,"threads_started":1}
I20260925 12:20:54.442305 18857 tablet_service.cc:1467] Processing CreateTablet for tablet 6836e8bdcd0d45b4b0bc4e99f4157dad (DEFAULT_TABLE table=test-table [id=c4a5a8397aaa4246b8ec4d469e3931e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:54.442613 18857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6836e8bdcd0d45b4b0bc4e99f4157dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:54.442576 18723 tablet_service.cc:1467] Processing CreateTablet for tablet 6836e8bdcd0d45b4b0bc4e99f4157dad (DEFAULT_TABLE table=test-table [id=c4a5a8397aaa4246b8ec4d469e3931e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:54.442869 18723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6836e8bdcd0d45b4b0bc4e99f4157dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:54.442948 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.438143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55084 (local address 127.0.98.1:32875)
0925 12:20:54.438312 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:54.438316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:54.439594 (+ 1278us) server_negotiation.cc:407] Connection header received
0925 12:20:54.439665 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:54.439667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:54.439684 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:54.439766 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:54.439996 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.440965 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.442034 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.442477 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.442559 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:54.442609 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:54.442857 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 12:20:54.442897 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
I20260925 12:20:54.443120 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.437872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32875 (local address 127.0.0.1:55084)
0925 12:20:54.439468 (+ 1596us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:54.439478 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:54.439529 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:54.439855 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:54.439856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:54.439858 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:54.439912 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:54.439914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.441057 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.441058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:54.441946 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:54.441949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.442694 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.442695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:54.442760 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:54.442764 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:20:54.443023 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 12:20:54.443065 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1583,"thread_start_us":47,"threads_started":1}
I20260925 12:20:54.443730 18589 tablet_service.cc:1467] Processing CreateTablet for tablet 6836e8bdcd0d45b4b0bc4e99f4157dad (DEFAULT_TABLE table=test-table [id=c4a5a8397aaa4246b8ec4d469e3931e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:54.444075 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6836e8bdcd0d45b4b0bc4e99f4157dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:54.446060 18937 tablet_bootstrap.cc:492] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6: Bootstrap starting.
I20260925 12:20:54.446342 18938 tablet_bootstrap.cc:492] T 6836e8bdcd0d45b4b0bc4e99f4157dad P f86a52e11072476699f18e221a390a15: Bootstrap starting.
I20260925 12:20:54.446817 18939 tablet_bootstrap.cc:492] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75: Bootstrap starting.
I20260925 12:20:54.446956 18938 tablet_bootstrap.cc:654] T 6836e8bdcd0d45b4b0bc4e99f4157dad P f86a52e11072476699f18e221a390a15: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:54.447072 18937 tablet_bootstrap.cc:654] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:54.447517 18939 tablet_bootstrap.cc:654] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:54.447528 18937 log.cc:824] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:54.447542 18938 log.cc:824] T 6836e8bdcd0d45b4b0bc4e99f4157dad P f86a52e11072476699f18e221a390a15: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:54.448115 18939 log.cc:824] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:54.448261 18938 tablet_bootstrap.cc:492] T 6836e8bdcd0d45b4b0bc4e99f4157dad P f86a52e11072476699f18e221a390a15: No bootstrap required, opened a new log
I20260925 12:20:54.448287 18937 tablet_bootstrap.cc:492] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6: No bootstrap required, opened a new log
I20260925 12:20:54.448334 18938 ts_tablet_manager.cc:1397] T 6836e8bdcd0d45b4b0bc4e99f4157dad P f86a52e11072476699f18e221a390a15: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:20:54.448369 18937 ts_tablet_manager.cc:1397] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:20:54.448850 18939 tablet_bootstrap.cc:492] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75: No bootstrap required, opened a new log
I20260925 12:20:54.448912 18939 ts_tablet_manager.cc:1397] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:20:54.449676 18937 raft_consensus.cc:348] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86a52e11072476699f18e221a390a15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33607 } } peers { permanent_uuid: "0dfc09dd231742a79967cc8b42321e75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32875 } } peers { permanent_uuid: "51f81649ddd8452bbe4a03b4eed9b7b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36247 } }
I20260925 12:20:54.449831 18937 raft_consensus.cc:374] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:54.449854 18937 raft_consensus.cc:729] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51f81649ddd8452bbe4a03b4eed9b7b6, State: Initialized, Role: FOLLOWER
I20260925 12:20:54.449975 18937 consensus_queue.cc:260] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86a52e11072476699f18e221a390a15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33607 } } peers { permanent_uuid: "0dfc09dd231742a79967cc8b42321e75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32875 } } peers { permanent_uuid: "51f81649ddd8452bbe4a03b4eed9b7b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36247 } }
I20260925 12:20:54.450244 18937 ts_tablet_manager.cc:1428] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:20:54.450255 18923 heartbeater.cc:503] Master 127.0.98.62:46165 was elected leader, sending a full tablet report...
I20260925 12:20:54.450484 18938 raft_consensus.cc:348] T 6836e8bdcd0d45b4b0bc4e99f4157dad P f86a52e11072476699f18e221a390a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86a52e11072476699f18e221a390a15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33607 } } peers { permanent_uuid: "0dfc09dd231742a79967cc8b42321e75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32875 } } peers { permanent_uuid: "51f81649ddd8452bbe4a03b4eed9b7b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36247 } }
I20260925 12:20:54.450639 18938 raft_consensus.cc:374] T 6836e8bdcd0d45b4b0bc4e99f4157dad P f86a52e11072476699f18e221a390a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:54.450670 18938 raft_consensus.cc:729] T 6836e8bdcd0d45b4b0bc4e99f4157dad P f86a52e11072476699f18e221a390a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86a52e11072476699f18e221a390a15, State: Initialized, Role: FOLLOWER
I20260925 12:20:54.450778 18938 consensus_queue.cc:260] T 6836e8bdcd0d45b4b0bc4e99f4157dad P f86a52e11072476699f18e221a390a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86a52e11072476699f18e221a390a15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33607 } } peers { permanent_uuid: "0dfc09dd231742a79967cc8b42321e75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32875 } } peers { permanent_uuid: "51f81649ddd8452bbe4a03b4eed9b7b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36247 } }
I20260925 12:20:54.450799 18939 raft_consensus.cc:348] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86a52e11072476699f18e221a390a15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33607 } } peers { permanent_uuid: "0dfc09dd231742a79967cc8b42321e75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32875 } } peers { permanent_uuid: "51f81649ddd8452bbe4a03b4eed9b7b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36247 } }
I20260925 12:20:54.450971 18939 raft_consensus.cc:374] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:54.451004 18939 raft_consensus.cc:729] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfc09dd231742a79967cc8b42321e75, State: Initialized, Role: FOLLOWER
I20260925 12:20:54.451059 18938 ts_tablet_manager.cc:1428] T 6836e8bdcd0d45b4b0bc4e99f4157dad P f86a52e11072476699f18e221a390a15: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:20:54.451076 18789 heartbeater.cc:503] Master 127.0.98.62:46165 was elected leader, sending a full tablet report...
I20260925 12:20:54.451103 18939 consensus_queue.cc:260] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86a52e11072476699f18e221a390a15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33607 } } peers { permanent_uuid: "0dfc09dd231742a79967cc8b42321e75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32875 } } peers { permanent_uuid: "51f81649ddd8452bbe4a03b4eed9b7b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36247 } }
I20260925 12:20:54.451340 18655 heartbeater.cc:503] Master 127.0.98.62:46165 was elected leader, sending a full tablet report...
I20260925 12:20:54.451340 18939 ts_tablet_manager.cc:1428] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260925 12:20:54.513293 18790 tablet.cc:2369] T 6836e8bdcd0d45b4b0bc4e99f4157dad P f86a52e11072476699f18e221a390a15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 12:20:54.624419 18656 tablet.cc:2369] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 12:20:54.663074 18924 tablet.cc:2369] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:20:54.664165 18945 raft_consensus.cc:482] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:20:54.664251 18945 raft_consensus.cc:504] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86a52e11072476699f18e221a390a15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33607 } } peers { permanent_uuid: "0dfc09dd231742a79967cc8b42321e75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32875 } } peers { permanent_uuid: "51f81649ddd8452bbe4a03b4eed9b7b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36247 } }
I20260925 12:20:54.664636 18945 leader_election.cc:288] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f86a52e11072476699f18e221a390a15 (127.0.98.2:33607), 51f81649ddd8452bbe4a03b4eed9b7b6 (127.0.98.3:36247)
I20260925 12:20:54.667495 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.664909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38761 (local address 127.0.98.3:36247)
0925 12:20:54.665008 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:54.665010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:54.665020 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:20:54.665047 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:54.665048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:54.665058 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:54.665140 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:54.665321 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.665794 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.666797 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.667146 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.667233 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:54.667242 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:54.667388 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 12:20:54.667423 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 12:20:54.667516 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.664779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36247 (local address 127.0.98.1:38761)
0925 12:20:54.664897 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:54.664906 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:54.664995 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:54.665144 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:54.665157 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:54.665159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:54.665235 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:54.665236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.665874 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.665875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:54.666702 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:54.666704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.667237 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.667238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:54.667316 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:54.667319 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:20:54.667383 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 12:20:54.667421 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":64,"thread_start_us":64,"threads_started":1}
I20260925 12:20:54.667510 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.664835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53349 (local address 127.0.98.2:33607)
0925 12:20:54.664882 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:54.664884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:54.664953 (+ 69us) server_negotiation.cc:407] Connection header received
0925 12:20:54.665083 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:54.665084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:54.665097 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:54.665181 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:54.665329 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.665788 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.666799 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.667152 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.667228 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:54.667239 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:54.667397 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 12:20:54.667439 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:20:54.667536 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.664774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33607 (local address 127.0.98.1:53349)
0925 12:20:54.664885 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:54.664898 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:54.664998 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:54.665184 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:54.665184 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:54.665186 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:54.665252 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:54.665254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.665877 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.665878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:54.666698 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:54.666700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.667239 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.667241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:54.667317 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:54.667329 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 12:20:54.667392 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 12:20:54.667430 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":37,"thread_start_us":63,"threads_started":1}
I20260925 12:20:54.667927 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6836e8bdcd0d45b4b0bc4e99f4157dad" candidate_uuid: "0dfc09dd231742a79967cc8b42321e75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51f81649ddd8452bbe4a03b4eed9b7b6" is_pre_election: true
I20260925 12:20:54.667989 18743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6836e8bdcd0d45b4b0bc4e99f4157dad" candidate_uuid: "0dfc09dd231742a79967cc8b42321e75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f86a52e11072476699f18e221a390a15" is_pre_election: true
I20260925 12:20:54.668079 18877 raft_consensus.cc:2454] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dfc09dd231742a79967cc8b42321e75 in term 0.
I20260925 12:20:54.668099 18743 raft_consensus.cc:2454] T 6836e8bdcd0d45b4b0bc4e99f4157dad P f86a52e11072476699f18e221a390a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dfc09dd231742a79967cc8b42321e75 in term 0.
I20260925 12:20:54.668365 18544 leader_election.cc:302] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfc09dd231742a79967cc8b42321e75, f86a52e11072476699f18e221a390a15; no voters:
I20260925 12:20:54.668521 18945 raft_consensus.cc:2781] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:20:54.668566 18945 raft_consensus.cc:482] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:20:54.668582 18945 raft_consensus.cc:3037] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:54.669190 18945 raft_consensus.cc:504] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86a52e11072476699f18e221a390a15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33607 } } peers { permanent_uuid: "0dfc09dd231742a79967cc8b42321e75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32875 } } peers { permanent_uuid: "51f81649ddd8452bbe4a03b4eed9b7b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36247 } }
I20260925 12:20:54.669322 18945 leader_election.cc:288] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [CANDIDATE]: Term 1 election: Requested vote from peers f86a52e11072476699f18e221a390a15 (127.0.98.2:33607), 51f81649ddd8452bbe4a03b4eed9b7b6 (127.0.98.3:36247)
I20260925 12:20:54.669488 18743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6836e8bdcd0d45b4b0bc4e99f4157dad" candidate_uuid: "0dfc09dd231742a79967cc8b42321e75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f86a52e11072476699f18e221a390a15"
I20260925 12:20:54.669484 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6836e8bdcd0d45b4b0bc4e99f4157dad" candidate_uuid: "0dfc09dd231742a79967cc8b42321e75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51f81649ddd8452bbe4a03b4eed9b7b6"
I20260925 12:20:54.669565 18743 raft_consensus.cc:3037] T 6836e8bdcd0d45b4b0bc4e99f4157dad P f86a52e11072476699f18e221a390a15 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:54.669585 18877 raft_consensus.cc:3037] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:54.670290 18743 raft_consensus.cc:2454] T 6836e8bdcd0d45b4b0bc4e99f4157dad P f86a52e11072476699f18e221a390a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dfc09dd231742a79967cc8b42321e75 in term 1.
I20260925 12:20:54.670303 18877 raft_consensus.cc:2454] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dfc09dd231742a79967cc8b42321e75 in term 1.
I20260925 12:20:54.670457 18544 leader_election.cc:302] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfc09dd231742a79967cc8b42321e75, f86a52e11072476699f18e221a390a15; no voters:
I20260925 12:20:54.670538 18945 raft_consensus.cc:2781] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:20:54.670684 18945 raft_consensus.cc:686] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [term 1 LEADER]: Becoming Leader. State: Replica: 0dfc09dd231742a79967cc8b42321e75, State: Running, Role: LEADER
I20260925 12:20:54.670775 18945 consensus_queue.cc:237] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86a52e11072476699f18e221a390a15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33607 } } peers { permanent_uuid: "0dfc09dd231742a79967cc8b42321e75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32875 } } peers { permanent_uuid: "51f81649ddd8452bbe4a03b4eed9b7b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36247 } }
I20260925 12:20:54.671526 18464 catalog_manager.cc:5495] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dfc09dd231742a79967cc8b42321e75 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0dfc09dd231742a79967cc8b42321e75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86a52e11072476699f18e221a390a15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfc09dd231742a79967cc8b42321e75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51f81649ddd8452bbe4a03b4eed9b7b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36247 } health_report { overall_health: UNKNOWN } } }
I20260925 12:20:54.724072 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.721974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55086 (local address 127.0.98.1:32875)
0925 12:20:54.722049 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:54.722051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:54.722059 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:20:54.722124 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:54.722125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:54.722141 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:54.722195 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:20:54.722318 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.722734 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.723390 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.723552 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.723598 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:54.723741 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:20:54.723819 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:20:54.723887 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:54.723941 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 12:20:54.723950 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 12:20:54.725672 18877 raft_consensus.cc:1261] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6 [term 1 FOLLOWER]: Refusing update from remote peer 0dfc09dd231742a79967cc8b42321e75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:20:54.725672 18743 raft_consensus.cc:1261] T 6836e8bdcd0d45b4b0bc4e99f4157dad P f86a52e11072476699f18e221a390a15 [term 1 FOLLOWER]: Refusing update from remote peer 0dfc09dd231742a79967cc8b42321e75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:20:54.726012 18945 consensus_queue.cc:1035] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f86a52e11072476699f18e221a390a15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:20:54.726140 18948 consensus_queue.cc:1035] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f81649ddd8452bbe4a03b4eed9b7b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:20:54.729043 18952 mvcc.cc:205] Tried to move back new op lower bound from 7333227948954091520 to 7333227948732055552. Current Snapshot: MvccSnapshot[applied={T|T < 7333227948954091520}]
I20260925 12:20:54.730090 18953 mvcc.cc:205] Tried to move back new op lower bound from 7333227948954091520 to 7333227948732055552. Current Snapshot: MvccSnapshot[applied={T|T < 7333227948954091520 or (T in {7333227948954091520})}]
I20260925 12:20:54.730351 18464 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:20:54.730430 18464 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:20:54.731168 18464 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 463bdc7ebf6f42c2baecda5a288f7b2a: Sending DeleteTablet for 3 replicas of tablet 6836e8bdcd0d45b4b0bc4e99f4157dad
I20260925 12:20:54.731462 18723 tablet_service.cc:1514] Processing DeleteTablet for tablet 6836e8bdcd0d45b4b0bc4e99f4157dad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:20:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46856
I20260925 12:20:54.731456 18857 tablet_service.cc:1514] Processing DeleteTablet for tablet 6836e8bdcd0d45b4b0bc4e99f4157dad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:20:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37814
I20260925 12:20:54.731518 18589 tablet_service.cc:1514] Processing DeleteTablet for tablet 6836e8bdcd0d45b4b0bc4e99f4157dad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:20:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55084
I20260925 12:20:54.731663 18961 tablet_replica.cc:330] T 6836e8bdcd0d45b4b0bc4e99f4157dad P f86a52e11072476699f18e221a390a15: stopping tablet replica
I20260925 12:20:54.731707 18963 tablet_replica.cc:330] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75: stopping tablet replica
I20260925 12:20:54.731746 18961 raft_consensus.cc:2229] T 6836e8bdcd0d45b4b0bc4e99f4157dad P f86a52e11072476699f18e221a390a15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:20:54.731804 18963 raft_consensus.cc:2229] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:20:54.731870 18961 raft_consensus.cc:2258] T 6836e8bdcd0d45b4b0bc4e99f4157dad P f86a52e11072476699f18e221a390a15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:20:54.732007 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 18526
I20260925 12:20:54.732043 18962 tablet_replica.cc:330] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6: stopping tablet replica
I20260925 12:20:54.732057 18963 raft_consensus.cc:2258] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 0dfc09dd231742a79967cc8b42321e75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:20:54.732112 18962 raft_consensus.cc:2229] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:20:54.732539 18962 raft_consensus.cc:2258] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:20:54.732937 18962 ts_tablet_manager.cc:1905] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:20:54.734745 18962 ts_tablet_manager.cc:1918] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:20:54.734818 18962 log.cc:1196] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/6836e8bdcd0d45b4b0bc4e99f4157dad
I20260925 12:20:54.735080 18962 ts_tablet_manager.cc:1939] T 6836e8bdcd0d45b4b0bc4e99f4157dad P 51f81649ddd8452bbe4a03b4eed9b7b6: Deleting consensus metadata
I20260925 12:20:54.735720 18449 catalog_manager.cc:4841] TS 51f81649ddd8452bbe4a03b4eed9b7b6 (127.0.98.3:36247): tablet 6836e8bdcd0d45b4b0bc4e99f4157dad (table test-table [id=c4a5a8397aaa4246b8ec4d469e3931e3]) successfully deleted
W20260925 12:20:54.738906 18450 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32875 (error 108)
I20260925 12:20:54.739207 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 18660
I20260925 12:20:54.739257 18961 ts_tablet_manager.cc:1905] T 6836e8bdcd0d45b4b0bc4e99f4157dad P f86a52e11072476699f18e221a390a15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:20:54.739800 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.739242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32875 (local address 127.0.0.1:55100)
0925 12:20:54.739713 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:54.739750 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452}
W20260925 12:20:54.739964 18450 catalog_manager.cc:4584] TS 0dfc09dd231742a79967cc8b42321e75 (127.0.98.1:32875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6836e8bdcd0d45b4b0bc4e99f4157dad: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32875: connect: Connection refused (error 111)
I20260925 12:20:54.744494 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.744389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33607 (local address 127.0.0.1:46868)
0925 12:20:54.744436 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:54.744464 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 12:20:54.744495 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 18794
I20260925 12:20:54.749085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 18432
2026-09-25T12:20:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2053 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T12:20:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:20:54Z Disabled control of system clock
I20260925 12:20:54.775178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45457
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 12:20:54.847929 18968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:54.848102 18968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:54.848122 18968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:54.849140 18968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:20:54.849169 18968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:54.849180 18968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:54.849191 18968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:20:54.849202 18968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:20:54.850371 18968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45457
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:54.850531 18968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:54.850728 18968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:54.855407 18974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:54.855384 18973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:54.855400 18976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:54.855571 18968 server_base.cc:1034] running on GCE node
I20260925 12:20:54.855808 18968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:54.856074 18968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:54.857205 18968 hybrid_clock.cc:648] HybridClock initialized: now 1790338854857184 us; error 37 us; skew 500 ppm
I20260925 12:20:54.863679 18968 webserver.cc:460] Webserver started at http://127.0.98.62:40757/ using document root <none> and password file <none>
I20260925 12:20:54.863898 18968 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:54.863953 18968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:54.864115 18968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:54.865079 18968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "a274671e17a24277be59a688481bad0f"
format_stamp: "Formatted at 2026-09-25 12:20:54 on dist-test-slave-clmx"
server_key: "1758d9e9ea2c97f1e807bd06aa4ec898"
server_key_iv: "bac22b644e8e0910ad7e7feb7a5d0cc1"
server_key_version: "encryptionkey@0"
I20260925 12:20:54.865445 18968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "a274671e17a24277be59a688481bad0f"
format_stamp: "Formatted at 2026-09-25 12:20:54 on dist-test-slave-clmx"
server_key: "1758d9e9ea2c97f1e807bd06aa4ec898"
server_key_iv: "bac22b644e8e0910ad7e7feb7a5d0cc1"
server_key_version: "encryptionkey@0"
I20260925 12:20:54.866612 18968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:20:54.867250 18982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:54.867417 18968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:20:54.867507 18968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "a274671e17a24277be59a688481bad0f"
format_stamp: "Formatted at 2026-09-25 12:20:54 on dist-test-slave-clmx"
server_key: "1758d9e9ea2c97f1e807bd06aa4ec898"
server_key_iv: "bac22b644e8e0910ad7e7feb7a5d0cc1"
server_key_version: "encryptionkey@0"
I20260925 12:20:54.867588 18968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:54.881214 18968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:54.881469 18968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:54.881593 18968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:54.885397 18968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45457
I20260925 12:20:54.885429 19034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45457 every 8 connection(s)
I20260925 12:20:54.885710 18968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:20:54.886536 19035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:54.888577 19035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f: Bootstrap starting.
I20260925 12:20:54.888949 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 18968
I20260925 12:20:54.889062 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:20:54.889200 19035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:54.889264 392 external_mini_cluster.cc:1428] Setting key 3d72f3c3c006bddbc22d972c8064e2b2
I20260925 12:20:54.889487 19035 log.cc:824] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:54.890231 19035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f: No bootstrap required, opened a new log
I20260925 12:20:54.892341 19035 raft_consensus.cc:348] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a274671e17a24277be59a688481bad0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45457 } }
I20260925 12:20:54.892524 19035 raft_consensus.cc:374] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:54.892553 19035 raft_consensus.cc:729] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a274671e17a24277be59a688481bad0f, State: Initialized, Role: FOLLOWER
I20260925 12:20:54.892656 19035 consensus_queue.cc:260] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a274671e17a24277be59a688481bad0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45457 } }
I20260925 12:20:54.892715 19035 raft_consensus.cc:388] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:20:54.892747 19035 raft_consensus.cc:482] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:20:54.892797 19035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:54.893298 19035 raft_consensus.cc:504] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a274671e17a24277be59a688481bad0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45457 } }
I20260925 12:20:54.893420 19035 leader_election.cc:302] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a274671e17a24277be59a688481bad0f; no voters:
I20260925 12:20:54.893581 19035 leader_election.cc:288] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:20:54.893604 19040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:20:54.893740 19040 raft_consensus.cc:686] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f [term 1 LEADER]: Becoming Leader. State: Replica: a274671e17a24277be59a688481bad0f, State: Running, Role: LEADER
I20260925 12:20:54.893817 19040 consensus_queue.cc:237] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a274671e17a24277be59a688481bad0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45457 } }
I20260925 12:20:54.893863 19035 sys_catalog.cc:564] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:20:54.894184 19042 sys_catalog.cc:455] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a274671e17a24277be59a688481bad0f. Latest consensus state: current_term: 1 leader_uuid: "a274671e17a24277be59a688481bad0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a274671e17a24277be59a688481bad0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45457 } } }
I20260925 12:20:54.894258 19042 sys_catalog.cc:458] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f [sys.catalog]: This master's current role is: LEADER
I20260925 12:20:54.894870 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:54.889715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.0.98.62:45457)
0925 12:20:54.889921 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:54.889929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:54.890006 (+ 77us) server_negotiation.cc:407] Connection header received
0925 12:20:54.890495 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:54.890501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:54.890555 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:54.890660 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:54.890915 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.891784 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.892937 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:54.893298 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:54.893407 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:54.894081 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:54.894084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:54.894085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:20:54.894271 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:54.894330 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:20:54.894346 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:54.894409 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:54.894596 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 12:20:54.894663 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260925 12:20:54.895063 19041 sys_catalog.cc:455] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a274671e17a24277be59a688481bad0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a274671e17a24277be59a688481bad0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45457 } } }
I20260925 12:20:54.895160 19041 sys_catalog.cc:458] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f [sys.catalog]: This master's current role is: LEADER
I20260925 12:20:54.895123 19046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:20:54.895628 19046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:20:54.897054 19046 catalog_manager.cc:1347] Generated new cluster ID: 11d3b293ad254e6bb15dbc6a0a019329
I20260925 12:20:54.897109 19046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:20:54.925251 19046 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:20:54.925765 19046 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:20:54.930253 19046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f: Generated new TSK 0
I20260925 12:20:54.930433 19046 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:20:54.932574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45457
--builtin_ntp_servers=127.0.98.20:39821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 12:20:55.008785 19059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:55.008967 19059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:55.008988 19059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:55.010087 19059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:55.010123 19059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:55.010146 19059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:20:55.011421 19059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:55.011652 19059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:55.011936 19059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:55.016376 19065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:55.016422 19067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:55.016454 19064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:55.016963 19059 server_base.cc:1034] running on GCE node
I20260925 12:20:55.017145 19059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:55.017380 19059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:55.018535 19059 hybrid_clock.cc:648] HybridClock initialized: now 1790338855018519 us; error 34 us; skew 500 ppm
I20260925 12:20:55.019702 19059 webserver.cc:460] Webserver started at http://127.0.98.1:42091/ using document root <none> and password file <none>
I20260925 12:20:55.019955 19059 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:55.020001 19059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:55.020136 19059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:55.021095 19059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "491e460918a849fd8cb05dda9c2e9803"
format_stamp: "Formatted at 2026-09-25 12:20:55 on dist-test-slave-clmx"
server_key: "340e3f0fcf901d714cc172c8871eb3cd"
server_key_iv: "ae643e90ef71de2ed1dc21fd506906ff"
server_key_version: "encryptionkey@0"
I20260925 12:20:55.021425 19059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "491e460918a849fd8cb05dda9c2e9803"
format_stamp: "Formatted at 2026-09-25 12:20:55 on dist-test-slave-clmx"
server_key: "340e3f0fcf901d714cc172c8871eb3cd"
server_key_iv: "ae643e90ef71de2ed1dc21fd506906ff"
server_key_version: "encryptionkey@0"
I20260925 12:20:55.022619 19059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:20:55.023262 19073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:55.023411 19059 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:20:55.023507 19059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "491e460918a849fd8cb05dda9c2e9803"
format_stamp: "Formatted at 2026-09-25 12:20:55 on dist-test-slave-clmx"
server_key: "340e3f0fcf901d714cc172c8871eb3cd"
server_key_iv: "ae643e90ef71de2ed1dc21fd506906ff"
server_key_version: "encryptionkey@0"
I20260925 12:20:55.023581 19059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:55.065296 19059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:55.065557 19059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:55.065699 19059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:55.065904 19059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:55.066195 19059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:55.066246 19059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:55.066275 19059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:55.066301 19059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:55.072078 19059 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43943
I20260925 12:20:55.072124 19186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43943 every 8 connection(s)
I20260925 12:20:55.072407 19059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:20:55.076761 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:55.073003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46593 (local address 127.0.98.62:45457)
0925 12:20:55.073101 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:55.073103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:55.073273 (+ 170us) server_negotiation.cc:407] Connection header received
0925 12:20:55.073713 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:55.073714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:55.073723 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:55.073769 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:55.074343 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.074795 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.075684 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.075868 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.075918 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:55.076554 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:55.076555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:55.076555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:20:55.076559 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:55.076570 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:20:55.076573 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:55.076614 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:55.076689 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 12:20:55.076707 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 12:20:55.076920 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:55.072966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45457 (local address 127.0.98.1:46593)
0925 12:20:55.073163 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:55.073193 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:55.073625 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:55.073836 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:55.073838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:55.073846 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:55.074277 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:55.074280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.074870 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.074871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:55.075564 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:55.075566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.075937 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.075937 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:55.076035 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:55.076439 (+ 404us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:20:55.076440 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:55.076463 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:20:55.076483 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:55.076625 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:55.076629 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:55.076631 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:20:55.076679 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 12:20:55.076745 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":116,"threads_started":1}
I20260925 12:20:55.077328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 19059
I20260925 12:20:55.077349 19187 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45457
I20260925 12:20:55.077428 19187 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:55.077433 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:20:55.077611 19187 heartbeater.cc:511] Master 127.0.98.62:45457 requested a full tablet report, sending...
I20260925 12:20:55.077620 392 external_mini_cluster.cc:1428] Setting key 1e241525e5ba375b66eb58e2ad3499e7
I20260925 12:20:55.078092 18999 ts_manager.cc:194] Registered new tserver with Master: 491e460918a849fd8cb05dda9c2e9803 (127.0.98.1:43943)
I20260925 12:20:55.078632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45457
--builtin_ntp_servers=127.0.98.20:39821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 12:20:55.078888 18999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46593
W20260925 12:20:55.156697 19190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:55.156870 19190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:55.156888 19190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:55.157907 19190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:55.157939 19190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:55.157960 19190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:20:55.159174 19190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:55.159435 19190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:55.159727 19190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:55.164186 19195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:55.164222 19198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:55.164194 19196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:55.164549 19190 server_base.cc:1034] running on GCE node
I20260925 12:20:55.164752 19190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:55.164978 19190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:55.166138 19190 hybrid_clock.cc:648] HybridClock initialized: now 1790338855166111 us; error 39 us; skew 500 ppm
I20260925 12:20:55.167280 19190 webserver.cc:460] Webserver started at http://127.0.98.2:33433/ using document root <none> and password file <none>
I20260925 12:20:55.167526 19190 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:55.167590 19190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:55.167738 19190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:55.168685 19190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "6b7453068ea64012b675e75ad74e3be2"
format_stamp: "Formatted at 2026-09-25 12:20:55 on dist-test-slave-clmx"
server_key: "bc3943388223b46966c12756e4a242af"
server_key_iv: "77c56f865174d3a04e4a0138624653af"
server_key_version: "encryptionkey@0"
I20260925 12:20:55.169061 19190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6b7453068ea64012b675e75ad74e3be2"
format_stamp: "Formatted at 2026-09-25 12:20:55 on dist-test-slave-clmx"
server_key: "bc3943388223b46966c12756e4a242af"
server_key_iv: "77c56f865174d3a04e4a0138624653af"
server_key_version: "encryptionkey@0"
I20260925 12:20:55.170321 19190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:20:55.171058 19204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:55.171233 19190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:20:55.171299 19190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "6b7453068ea64012b675e75ad74e3be2"
format_stamp: "Formatted at 2026-09-25 12:20:55 on dist-test-slave-clmx"
server_key: "bc3943388223b46966c12756e4a242af"
server_key_iv: "77c56f865174d3a04e4a0138624653af"
server_key_version: "encryptionkey@0"
I20260925 12:20:55.171389 19190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:55.196069 19190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:55.196305 19190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:55.196442 19190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:55.196636 19190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:55.196928 19190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:55.196979 19190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:55.197009 19190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:55.197036 19190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:55.202157 19190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45317
I20260925 12:20:55.202190 19317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45317 every 8 connection(s)
I20260925 12:20:55.202485 19190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:20:55.204478 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 19190
I20260925 12:20:55.204591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:20:55.204784 392 external_mini_cluster.cc:1428] Setting key 96136912a8099e434ceb0d7cce886885
I20260925 12:20:55.205775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45457
--builtin_ntp_servers=127.0.98.20:39821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 12:20:55.208057 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:55.203150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34999 (local address 127.0.98.62:45457)
0925 12:20:55.203211 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:55.203212 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:55.203402 (+ 190us) server_negotiation.cc:407] Connection header received
0925 12:20:55.203917 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:55.203918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:55.203927 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:55.203987 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:55.204791 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.205416 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.206807 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.206985 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.207059 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:55.207806 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:55.207808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:55.207808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:20:55.207812 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:55.207825 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:20:55.207836 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:55.207883 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:55.207962 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 12:20:55.207985 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 12:20:55.208289 19320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:55.203127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45457 (local address 127.0.98.2:34999)
0925 12:20:55.203293 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:55.203328 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:55.203817 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:55.204053 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:55.204056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:55.204063 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:55.204700 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:55.204707 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.205483 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.205485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:55.206719 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:55.206722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.207068 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.207069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:55.207187 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:55.207671 (+ 484us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:20:55.207672 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:55.207696 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:20:55.207716 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:55.207882 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:55.207887 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:55.207890 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:20:55.207966 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 12:20:55.208054 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260925 12:20:55.208817 19318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45457
I20260925 12:20:55.208894 19318 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:55.209064 19318 heartbeater.cc:511] Master 127.0.98.62:45457 requested a full tablet report, sending...
I20260925 12:20:55.209467 18999 ts_manager.cc:194] Registered new tserver with Master: 6b7453068ea64012b675e75ad74e3be2 (127.0.98.2:45317)
I20260925 12:20:55.210001 18999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34999
W20260925 12:20:55.282346 19321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:55.282516 19321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:55.282536 19321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:55.283566 19321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:55.283595 19321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:55.283672 19321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:20:55.284963 19321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:55.285137 19321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:55.285401 19321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:55.289975 19327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:55.290022 19329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:55.290046 19321 server_base.cc:1034] running on GCE node
W20260925 12:20:55.289983 19326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:55.290390 19321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:55.290617 19321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:55.291746 19321 hybrid_clock.cc:648] HybridClock initialized: now 1790338855291732 us; error 30 us; skew 500 ppm
I20260925 12:20:55.292816 19321 webserver.cc:460] Webserver started at http://127.0.98.3:44067/ using document root <none> and password file <none>
I20260925 12:20:55.292986 19321 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:55.293040 19321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:55.293120 19321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:55.294133 19321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "89371a4b49ff4285a0eeb69152d15a7b"
format_stamp: "Formatted at 2026-09-25 12:20:55 on dist-test-slave-clmx"
server_key: "e2fc514f81f2c6fa1a6b4b2365f468ef"
server_key_iv: "5ba8a430635ba3cc700ce2c907cb651b"
server_key_version: "encryptionkey@0"
I20260925 12:20:55.294504 19321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "89371a4b49ff4285a0eeb69152d15a7b"
format_stamp: "Formatted at 2026-09-25 12:20:55 on dist-test-slave-clmx"
server_key: "e2fc514f81f2c6fa1a6b4b2365f468ef"
server_key_iv: "5ba8a430635ba3cc700ce2c907cb651b"
server_key_version: "encryptionkey@0"
I20260925 12:20:55.295809 19321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:20:55.296429 19335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:55.296604 19321 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:20:55.296679 19321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "89371a4b49ff4285a0eeb69152d15a7b"
format_stamp: "Formatted at 2026-09-25 12:20:55 on dist-test-slave-clmx"
server_key: "e2fc514f81f2c6fa1a6b4b2365f468ef"
server_key_iv: "5ba8a430635ba3cc700ce2c907cb651b"
server_key_version: "encryptionkey@0"
I20260925 12:20:55.296729 19321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:55.307482 19321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:55.307747 19321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:55.307842 19321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:55.307986 19321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:55.308210 19321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:55.308238 19321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:55.308256 19321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:55.308269 19321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:55.313864 19321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41497
I20260925 12:20:55.313933 19448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41497 every 8 connection(s)
I20260925 12:20:55.314260 19321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:20:55.319121 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:55.314911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52093 (local address 127.0.98.62:45457)
0925 12:20:55.314990 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:55.314992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:55.315173 (+ 181us) server_negotiation.cc:407] Connection header received
0925 12:20:55.315677 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:55.315678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:55.315691 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:55.315759 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:55.316515 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.316979 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.317994 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.318154 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.318199 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:55.318884 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:55.318885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:55.318885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:20:55.318889 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:55.318901 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:20:55.318905 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:55.318943 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:55.319048 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 12:20:55.319066 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 12:20:55.319322 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:55.314878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45457 (local address 127.0.98.3:52093)
0925 12:20:55.315054 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:55.315084 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:55.315550 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:55.315826 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:55.315830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:55.315840 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:55.316433 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:55.316439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.317063 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.317065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:55.317926 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:55.317928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.318238 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.318239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:55.318345 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:55.318764 (+ 419us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:20:55.318765 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:55.318789 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:20:55.318810 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:55.318963 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:55.318967 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:55.318969 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:20:55.319038 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 12:20:55.319110 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":98,"threads_started":1}
I20260925 12:20:55.319841 19449 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45457
I20260925 12:20:55.319948 19449 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:55.320129 19449 heartbeater.cc:511] Master 127.0.98.62:45457 requested a full tablet report, sending...
I20260925 12:20:55.320492 18999 ts_manager.cc:194] Registered new tserver with Master: 89371a4b49ff4285a0eeb69152d15a7b (127.0.98.3:41497)
I20260925 12:20:55.320874 18999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:52093
I20260925 12:20:55.320943 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 19321
I20260925 12:20:55.321069 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:20:55.321259 392 external_mini_cluster.cc:1428] Setting key c8d67b65abd8ecd0304161094fde42c5
I20260925 12:20:55.321986 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:20:55.325968 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:55.323407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.0.98.62:45457)
0925 12:20:55.323455 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:55.323457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:55.323557 (+ 100us) server_negotiation.cc:407] Connection header received
0925 12:20:55.323660 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:55.323661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:55.323670 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:55.323724 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:55.323886 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.324304 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.324983 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.325157 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.325205 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:55.325731 (+ 526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:55.325732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:55.325732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:20:55.325735 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:55.325746 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:20:55.325750 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:55.325827 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:55.325914 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 12:20:55.325922 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 12:20:55.327595 18999 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:20:55.329141 18999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:20:55.334965 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:55.331158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41497 (local address 127.0.0.1:43206)
0925 12:20:55.331307 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:55.331316 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:55.331362 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:55.331781 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:55.331781 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:55.331783 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:55.331835 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:55.331836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.333058 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.333059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:55.334004 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:55.334007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.334770 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.334770 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:55.334828 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:55.334837 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 12:20:55.334884 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 12:20:55.334917 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":77,"spinlock_wait_cycles":768,"thread_start_us":72,"threads_started":1}
I20260925 12:20:55.335018 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:55.331464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43206 (local address 127.0.98.3:41497)
0925 12:20:55.331651 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:55.331655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:55.331665 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:20:55.331695 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:55.331697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:55.331716 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:55.331777 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:55.331920 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.332981 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.334085 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.334693 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.334765 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:55.334837 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:55.334913 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 12:20:55.334948 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
I20260925 12:20:55.335602 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:55.331350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45317 (local address 127.0.0.1:35954)
0925 12:20:55.331649 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:55.331658 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:55.331698 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:55.332101 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:55.332102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:55.332104 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:55.332144 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:55.332145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.332954 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.332955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:55.334255 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:55.334257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.335320 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.335321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:55.335380 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:55.335384 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:20:55.335492 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 12:20:55.335530 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":42,"thread_start_us":63,"threads_started":1}
I20260925 12:20:55.335676 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:55.331481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35954 (local address 127.0.98.2:45317)
0925 12:20:55.331893 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:55.331897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:55.331908 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:20:55.331946 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:55.331948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:55.331963 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:55.332034 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:55.332200 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.332855 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.334343 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.335058 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.335126 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:55.335213 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:55.335512 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 12:20:55.335551 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":121,"threads_started":1}
I20260925 12:20:55.335779 19383 tablet_service.cc:1467] Processing CreateTablet for tablet d18e2f07fe924485a74d6a7e004cc24c (DEFAULT_TABLE table=test-table [id=c47bb0ae5c5a4a8c9fefcebd512ea4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:55.336129 19383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d18e2f07fe924485a74d6a7e004cc24c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:55.336495 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:55.331158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43943 (local address 127.0.0.1:43148)
0925 12:20:55.331263 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:55.331298 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:55.331360 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:55.331749 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:55.331752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:55.331754 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:55.331852 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:55.331853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.333134 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.333135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:55.334259 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:55.334261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.336245 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.336246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:55.336308 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:55.336311 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:20:55.336370 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 12:20:55.336406 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":6,"thread_start_us":61,"threads_started":1}
I20260925 12:20:55.336520 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:55.331251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.0.98.1:43943)
0925 12:20:55.331491 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:55.331496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:55.331507 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:20:55.331549 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:55.331551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:55.331564 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:55.331677 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:55.331929 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.332984 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.335247 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.336012 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.336100 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:55.336205 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:55.336398 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 12:20:55.336443 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":91,"threads_started":1}
I20260925 12:20:55.336627 19252 tablet_service.cc:1467] Processing CreateTablet for tablet d18e2f07fe924485a74d6a7e004cc24c (DEFAULT_TABLE table=test-table [id=c47bb0ae5c5a4a8c9fefcebd512ea4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:55.336931 19252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d18e2f07fe924485a74d6a7e004cc24c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:55.338755 19463 tablet_bootstrap.cc:492] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b: Bootstrap starting.
I20260925 12:20:55.339289 19463 tablet_bootstrap.cc:654] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:55.339397 19464 tablet_bootstrap.cc:492] T d18e2f07fe924485a74d6a7e004cc24c P 6b7453068ea64012b675e75ad74e3be2: Bootstrap starting.
I20260925 12:20:55.339624 19463 log.cc:824] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:55.340060 19464 tablet_bootstrap.cc:654] T d18e2f07fe924485a74d6a7e004cc24c P 6b7453068ea64012b675e75ad74e3be2: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:55.340014 19121 tablet_service.cc:1467] Processing CreateTablet for tablet d18e2f07fe924485a74d6a7e004cc24c (DEFAULT_TABLE table=test-table [id=c47bb0ae5c5a4a8c9fefcebd512ea4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:55.340327 19121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d18e2f07fe924485a74d6a7e004cc24c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:55.340392 19464 log.cc:824] T d18e2f07fe924485a74d6a7e004cc24c P 6b7453068ea64012b675e75ad74e3be2: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:55.340507 19463 tablet_bootstrap.cc:492] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b: No bootstrap required, opened a new log
I20260925 12:20:55.340576 19463 ts_tablet_manager.cc:1397] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:20:55.341107 19464 tablet_bootstrap.cc:492] T d18e2f07fe924485a74d6a7e004cc24c P 6b7453068ea64012b675e75ad74e3be2: No bootstrap required, opened a new log
I20260925 12:20:55.341171 19464 ts_tablet_manager.cc:1397] T d18e2f07fe924485a74d6a7e004cc24c P 6b7453068ea64012b675e75ad74e3be2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:20:55.342244 19463 raft_consensus.cc:348] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491e460918a849fd8cb05dda9c2e9803" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 } } peers { permanent_uuid: "89371a4b49ff4285a0eeb69152d15a7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41497 } } peers { permanent_uuid: "6b7453068ea64012b675e75ad74e3be2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45317 } }
I20260925 12:20:55.342350 19463 raft_consensus.cc:374] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:55.342367 19463 raft_consensus.cc:729] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89371a4b49ff4285a0eeb69152d15a7b, State: Initialized, Role: FOLLOWER
I20260925 12:20:55.342473 19463 consensus_queue.cc:260] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491e460918a849fd8cb05dda9c2e9803" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 } } peers { permanent_uuid: "89371a4b49ff4285a0eeb69152d15a7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41497 } } peers { permanent_uuid: "6b7453068ea64012b675e75ad74e3be2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45317 } }
I20260925 12:20:55.342700 19463 ts_tablet_manager.cc:1428] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:20:55.342720 19449 heartbeater.cc:503] Master 127.0.98.62:45457 was elected leader, sending a full tablet report...
I20260925 12:20:55.343446 19468 tablet_bootstrap.cc:492] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803: Bootstrap starting.
I20260925 12:20:55.343482 19464 raft_consensus.cc:348] T d18e2f07fe924485a74d6a7e004cc24c P 6b7453068ea64012b675e75ad74e3be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491e460918a849fd8cb05dda9c2e9803" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 } } peers { permanent_uuid: "89371a4b49ff4285a0eeb69152d15a7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41497 } } peers { permanent_uuid: "6b7453068ea64012b675e75ad74e3be2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45317 } }
I20260925 12:20:55.343652 19464 raft_consensus.cc:374] T d18e2f07fe924485a74d6a7e004cc24c P 6b7453068ea64012b675e75ad74e3be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:55.343684 19464 raft_consensus.cc:729] T d18e2f07fe924485a74d6a7e004cc24c P 6b7453068ea64012b675e75ad74e3be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7453068ea64012b675e75ad74e3be2, State: Initialized, Role: FOLLOWER
I20260925 12:20:55.343786 19464 consensus_queue.cc:260] T d18e2f07fe924485a74d6a7e004cc24c P 6b7453068ea64012b675e75ad74e3be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491e460918a849fd8cb05dda9c2e9803" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 } } peers { permanent_uuid: "89371a4b49ff4285a0eeb69152d15a7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41497 } } peers { permanent_uuid: "6b7453068ea64012b675e75ad74e3be2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45317 } }
I20260925 12:20:55.343969 19464 ts_tablet_manager.cc:1428] T d18e2f07fe924485a74d6a7e004cc24c P 6b7453068ea64012b675e75ad74e3be2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:20:55.344046 19318 heartbeater.cc:503] Master 127.0.98.62:45457 was elected leader, sending a full tablet report...
I20260925 12:20:55.344132 19468 tablet_bootstrap.cc:654] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:55.344472 19468 log.cc:824] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:55.345274 19468 tablet_bootstrap.cc:492] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803: No bootstrap required, opened a new log
I20260925 12:20:55.345326 19468 ts_tablet_manager.cc:1397] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:20:55.346935 19468 raft_consensus.cc:348] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491e460918a849fd8cb05dda9c2e9803" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 } } peers { permanent_uuid: "89371a4b49ff4285a0eeb69152d15a7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41497 } } peers { permanent_uuid: "6b7453068ea64012b675e75ad74e3be2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45317 } }
I20260925 12:20:55.347039 19468 raft_consensus.cc:374] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:55.347059 19468 raft_consensus.cc:729] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491e460918a849fd8cb05dda9c2e9803, State: Initialized, Role: FOLLOWER
I20260925 12:20:55.347164 19468 consensus_queue.cc:260] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491e460918a849fd8cb05dda9c2e9803" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 } } peers { permanent_uuid: "89371a4b49ff4285a0eeb69152d15a7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41497 } } peers { permanent_uuid: "6b7453068ea64012b675e75ad74e3be2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45317 } }
I20260925 12:20:55.347397 19468 ts_tablet_manager.cc:1428] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:20:55.347426 19187 heartbeater.cc:503] Master 127.0.98.62:45457 was elected leader, sending a full tablet report...
W20260925 12:20:55.453001 19319 tablet.cc:2369] T d18e2f07fe924485a74d6a7e004cc24c P 6b7453068ea64012b675e75ad74e3be2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:20:55.509640 19471 raft_consensus.cc:482] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:20:55.509755 19471 raft_consensus.cc:504] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491e460918a849fd8cb05dda9c2e9803" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 } } peers { permanent_uuid: "89371a4b49ff4285a0eeb69152d15a7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41497 } } peers { permanent_uuid: "6b7453068ea64012b675e75ad74e3be2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45317 } }
I20260925 12:20:55.510077 19471 leader_election.cc:288] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89371a4b49ff4285a0eeb69152d15a7b (127.0.98.3:41497), 6b7453068ea64012b675e75ad74e3be2 (127.0.98.2:45317)
I20260925 12:20:55.512861 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:55.510360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58817 (local address 127.0.98.3:41497)
0925 12:20:55.510428 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:55.510430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:55.510443 (+ 13us) server_negotiation.cc:407] Connection header received
0925 12:20:55.510476 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:55.510477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:55.510487 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:55.510583 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:55.510771 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.511275 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.512185 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.512528 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.512609 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:55.512618 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:55.512762 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 12:20:55.512805 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 12:20:55.512851 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:55.510282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41497 (local address 127.0.98.1:58817)
0925 12:20:55.510351 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:55.510362 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:55.510407 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:55.510618 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:55.510619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:55.510621 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:55.510693 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:55.510695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.511356 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.511357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:55.512092 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:55.512094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.512609 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.512610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:55.512682 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:55.512685 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:20:55.512745 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 12:20:55.512783 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 12:20:55.513132 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:55.510515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56973 (local address 127.0.98.2:45317)
0925 12:20:55.510581 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:55.510583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:55.510645 (+ 62us) server_negotiation.cc:407] Connection header received
0925 12:20:55.510710 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:55.510711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:55.510720 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:55.510793 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:55.510953 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.511438 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.512389 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.512730 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.512807 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:55.512815 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:55.513021 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 12:20:55.513068 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 12:20:55.513211 19403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d18e2f07fe924485a74d6a7e004cc24c" candidate_uuid: "491e460918a849fd8cb05dda9c2e9803" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89371a4b49ff4285a0eeb69152d15a7b" is_pre_election: true
I20260925 12:20:55.513342 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:55.510400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45317 (local address 127.0.98.1:56973)
0925 12:20:55.510567 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:55.510576 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:55.510650 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:55.510797 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:55.510797 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:55.510799 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:55.510882 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:55.510883 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.511519 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.511520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:55.512294 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:55.512297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.512828 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.512829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:55.512925 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:55.512930 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:20:55.513212 (+ 282us) client_negotiation.cc:240] Negotiation successful
0925 12:20:55.513262 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":108,"threads_started":1}
I20260925 12:20:55.513378 19403 raft_consensus.cc:2454] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491e460918a849fd8cb05dda9c2e9803 in term 0.
I20260925 12:20:55.513622 19076 leader_election.cc:302] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491e460918a849fd8cb05dda9c2e9803, 89371a4b49ff4285a0eeb69152d15a7b; no voters:
I20260925 12:20:55.513777 19471 raft_consensus.cc:2781] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:20:55.513839 19471 raft_consensus.cc:482] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:20:55.513777 19272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d18e2f07fe924485a74d6a7e004cc24c" candidate_uuid: "491e460918a849fd8cb05dda9c2e9803" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7453068ea64012b675e75ad74e3be2" is_pre_election: true
I20260925 12:20:55.513862 19471 raft_consensus.cc:3037] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:55.513928 19272 raft_consensus.cc:2454] T d18e2f07fe924485a74d6a7e004cc24c P 6b7453068ea64012b675e75ad74e3be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491e460918a849fd8cb05dda9c2e9803 in term 0.
I20260925 12:20:55.514642 19471 raft_consensus.cc:504] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491e460918a849fd8cb05dda9c2e9803" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 } } peers { permanent_uuid: "89371a4b49ff4285a0eeb69152d15a7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41497 } } peers { permanent_uuid: "6b7453068ea64012b675e75ad74e3be2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45317 } }
I20260925 12:20:55.514762 19471 leader_election.cc:288] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803 [CANDIDATE]: Term 1 election: Requested vote from peers 89371a4b49ff4285a0eeb69152d15a7b (127.0.98.3:41497), 6b7453068ea64012b675e75ad74e3be2 (127.0.98.2:45317)
I20260925 12:20:55.514926 19403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d18e2f07fe924485a74d6a7e004cc24c" candidate_uuid: "491e460918a849fd8cb05dda9c2e9803" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89371a4b49ff4285a0eeb69152d15a7b"
I20260925 12:20:55.514947 19272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d18e2f07fe924485a74d6a7e004cc24c" candidate_uuid: "491e460918a849fd8cb05dda9c2e9803" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7453068ea64012b675e75ad74e3be2"
I20260925 12:20:55.514990 19403 raft_consensus.cc:3037] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:55.515079 19272 raft_consensus.cc:3037] T d18e2f07fe924485a74d6a7e004cc24c P 6b7453068ea64012b675e75ad74e3be2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:55.515821 19403 raft_consensus.cc:2454] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491e460918a849fd8cb05dda9c2e9803 in term 1.
I20260925 12:20:55.515883 19272 raft_consensus.cc:2454] T d18e2f07fe924485a74d6a7e004cc24c P 6b7453068ea64012b675e75ad74e3be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491e460918a849fd8cb05dda9c2e9803 in term 1.
I20260925 12:20:55.516057 19076 leader_election.cc:302] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491e460918a849fd8cb05dda9c2e9803, 89371a4b49ff4285a0eeb69152d15a7b; no voters:
I20260925 12:20:55.516142 19471 raft_consensus.cc:2781] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:20:55.516288 19471 raft_consensus.cc:686] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803 [term 1 LEADER]: Becoming Leader. State: Replica: 491e460918a849fd8cb05dda9c2e9803, State: Running, Role: LEADER
I20260925 12:20:55.516389 19471 consensus_queue.cc:237] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491e460918a849fd8cb05dda9c2e9803" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 } } peers { permanent_uuid: "89371a4b49ff4285a0eeb69152d15a7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41497 } } peers { permanent_uuid: "6b7453068ea64012b675e75ad74e3be2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45317 } }
I20260925 12:20:55.517132 18999 catalog_manager.cc:5495] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803 reported cstate change: term changed from 0 to 1, leader changed from <none> to 491e460918a849fd8cb05dda9c2e9803 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "491e460918a849fd8cb05dda9c2e9803" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491e460918a849fd8cb05dda9c2e9803" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89371a4b49ff4285a0eeb69152d15a7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b7453068ea64012b675e75ad74e3be2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20260925 12:20:55.522401 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:55.519696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.0.98.1:43943)
0925 12:20:55.519735 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:55.519737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:55.519744 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:20:55.519772 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:55.519773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:55.519789 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:55.519862 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:20:55.519985 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.520439 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.521645 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:55.521844 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:55.521914 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:55.522057 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:20:55.522150 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:20:55.522251 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:55.522280 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 12:20:55.522290 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:20:55.524013 19272 raft_consensus.cc:1261] T d18e2f07fe924485a74d6a7e004cc24c P 6b7453068ea64012b675e75ad74e3be2 [term 1 FOLLOWER]: Refusing update from remote peer 491e460918a849fd8cb05dda9c2e9803: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:20:55.524032 19403 raft_consensus.cc:1261] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b [term 1 FOLLOWER]: Refusing update from remote peer 491e460918a849fd8cb05dda9c2e9803: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:20:55.524314 19473 consensus_queue.cc:1035] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7453068ea64012b675e75ad74e3be2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:20:55.524426 19471 consensus_queue.cc:1035] T d18e2f07fe924485a74d6a7e004cc24c P 491e460918a849fd8cb05dda9c2e9803 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89371a4b49ff4285a0eeb69152d15a7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:20:55.527293 19478 mvcc.cc:205] Tried to move back new op lower bound from 7333227952223641600 to 7333227952195579904. Current Snapshot: MvccSnapshot[applied={T|T < 7333227952223641600}]
I20260925 12:20:55.528652 18999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:20:55.528735 18999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:20:55.529443 18999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a274671e17a24277be59a688481bad0f: Sending DeleteTablet for 3 replicas of tablet d18e2f07fe924485a74d6a7e004cc24c
I20260925 12:20:55.529770 19383 tablet_service.cc:1514] Processing DeleteTablet for tablet d18e2f07fe924485a74d6a7e004cc24c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:20:55 UTC) from {username='slave'} at 127.0.0.1:43206
I20260925 12:20:55.529943 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 19059
I20260925 12:20:55.530141 19252 tablet_service.cc:1514] Processing DeleteTablet for tablet d18e2f07fe924485a74d6a7e004cc24c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:20:55 UTC) from {username='slave'} at 127.0.0.1:35954
I20260925 12:20:55.530516 19485 tablet_replica.cc:330] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b: stopping tablet replica
I20260925 12:20:55.530582 19485 raft_consensus.cc:2229] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:20:55.530644 19485 raft_consensus.cc:2258] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:20:55.531205 19485 ts_tablet_manager.cc:1905] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:20:55.532781 19485 ts_tablet_manager.cc:1918] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:20:55.532863 19485 log.cc:1196] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/d18e2f07fe924485a74d6a7e004cc24c
I20260925 12:20:55.533105 19485 ts_tablet_manager.cc:1939] T d18e2f07fe924485a74d6a7e004cc24c P 89371a4b49ff4285a0eeb69152d15a7b: Deleting consensus metadata
I20260925 12:20:55.533476 18985 catalog_manager.cc:4841] TS 89371a4b49ff4285a0eeb69152d15a7b (127.0.98.3:41497): tablet d18e2f07fe924485a74d6a7e004cc24c (table test-table [id=c47bb0ae5c5a4a8c9fefcebd512ea4e3]) successfully deleted
W20260925 12:20:55.535267 18983 connection.cc:535] client connection to 127.0.98.1:43943 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 12:20:55.535351 18983 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 12:20:55.535894 19457 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:20:55.535658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43943 (local address 127.0.0.1:43162)
0925 12:20:55.535694 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:55.535705 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:55.535746 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:55.535856 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43943: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":18}
W20260925 12:20:55.536002 18983 catalog_manager.cc:4584] TS 491e460918a849fd8cb05dda9c2e9803 (127.0.98.1:43943): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d18e2f07fe924485a74d6a7e004cc24c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43943: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 12:20:55.536080 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 19190
I20260925 12:20:55.536103 19486 tablet_replica.cc:330] T d18e2f07fe924485a74d6a7e004cc24c P 6b7453068ea64012b675e75ad74e3be2: stopping tablet replica
I20260925 12:20:55.536185 19486 raft_consensus.cc:2229] T d18e2f07fe924485a74d6a7e004cc24c P 6b7453068ea64012b675e75ad74e3be2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:20:55.536270 19486 raft_consensus.cc:2258] T d18e2f07fe924485a74d6a7e004cc24c P 6b7453068ea64012b675e75ad74e3be2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:20:55.541149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 19321
I20260925 12:20:55.541167 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:55.541049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45317 (local address 127.0.0.1:35960)
0925 12:20:55.541100 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:55.541136 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 12:20:55.545399 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 18968
2026-09-25T12:20:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (795 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:20:55 dist-test-slave-clmx krb5kdc[19490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:20:55 dist-test-slave-clmx krb5kdc[19490](info): set up 2 sockets
Sep 25 12:20:55 dist-test-slave-clmx krb5kdc[19490](info): commencing operation
krb5kdc: starting...
W20260925 12:20:56.535331 392 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:20:56 dist-test-slave-clmx krb5kdc[19490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:20:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:20:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:20:56.642340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38463
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:20:56.717489 19503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:56.717677 19503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:56.717697 19503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:56.718842 19503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:20:56.718879 19503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:56.718894 19503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:56.718933 19503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:20:56.718961 19503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:20:56.720356 19503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:56.720582 19503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:56.720804 19503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:20:56.725476 19503 server_base.cc:1034] running on GCE node
W20260925 12:20:56.725449 19511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:56.725462 19509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:56.725461 19508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:56.725886 19503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:56.726202 19503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:56.727355 19503 hybrid_clock.cc:648] HybridClock initialized: now 1790338856727342 us; error 26 us; skew 500 ppm
Sep 25 12:20:56 dist-test-slave-clmx krb5kdc[19490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:56.729524 19503 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:20:56.729863 19503 webserver.cc:460] Webserver started at http://127.0.98.62:38987/ using document root <none> and password file <none>
I20260925 12:20:56.730093 19503 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:56.730157 19503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:56.730301 19503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:56.731267 19503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "9f18c840584641dfa9beb98c7dc83e16"
format_stamp: "Formatted at 2026-09-25 12:20:56 on dist-test-slave-clmx"
server_key: "9091ca0a334a660794c0e0073fc78cff"
server_key_iv: "5d2c615d9194abef337679be390f6e17"
server_key_version: "encryptionkey@0"
I20260925 12:20:56.731606 19503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "9f18c840584641dfa9beb98c7dc83e16"
format_stamp: "Formatted at 2026-09-25 12:20:56 on dist-test-slave-clmx"
server_key: "9091ca0a334a660794c0e0073fc78cff"
server_key_iv: "5d2c615d9194abef337679be390f6e17"
server_key_version: "encryptionkey@0"
I20260925 12:20:56.732877 19503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:20:56.733525 19518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:56.733678 19503 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:20:56.733789 19503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "9f18c840584641dfa9beb98c7dc83e16"
format_stamp: "Formatted at 2026-09-25 12:20:56 on dist-test-slave-clmx"
server_key: "9091ca0a334a660794c0e0073fc78cff"
server_key_iv: "5d2c615d9194abef337679be390f6e17"
server_key_version: "encryptionkey@0"
I20260925 12:20:56.733867 19503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:56.745467 19503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:56.747092 19503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:56.747275 19503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:56.750942 19503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38463
I20260925 12:20:56.750972 19570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38463 every 8 connection(s)
I20260925 12:20:56.751302 19503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:20:56.752135 19571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:56.754026 19571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16: Bootstrap starting.
I20260925 12:20:56.754531 19571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:56.754740 19571 log.cc:824] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:56.755304 19571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16: No bootstrap required, opened a new log
I20260925 12:20:56.756197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 19503
I20260925 12:20:56.756318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:20:56.756493 392 external_mini_cluster.cc:1428] Setting key babbe02019604c2dbeeaca2d15eda6d5
I20260925 12:20:56.757148 19571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f18c840584641dfa9beb98c7dc83e16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38463 } }
I20260925 12:20:56.757288 19571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:56.757341 19571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f18c840584641dfa9beb98c7dc83e16, State: Initialized, Role: FOLLOWER
I20260925 12:20:56.757457 19571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f18c840584641dfa9beb98c7dc83e16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38463 } }
I20260925 12:20:56.757542 19571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:20:56.757591 19571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:20:56.757635 19571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:56.758100 19571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f18c840584641dfa9beb98c7dc83e16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38463 } }
I20260925 12:20:56.758208 19571 leader_election.cc:302] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f18c840584641dfa9beb98c7dc83e16; no voters:
I20260925 12:20:56.758338 19571 leader_election.cc:288] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:20:56.758366 19576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:20:56.758460 19576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16 [term 1 LEADER]: Becoming Leader. State: Replica: 9f18c840584641dfa9beb98c7dc83e16, State: Running, Role: LEADER
I20260925 12:20:56.758554 19571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:20:56.758544 19576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f18c840584641dfa9beb98c7dc83e16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38463 } }
I20260925 12:20:56.758852 19576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f18c840584641dfa9beb98c7dc83e16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f18c840584641dfa9beb98c7dc83e16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38463 } } }
I20260925 12:20:56.758993 19576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16 [sys.catalog]: This master's current role is: LEADER
I20260925 12:20:56.759159 19576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f18c840584641dfa9beb98c7dc83e16. Latest consensus state: current_term: 1 leader_uuid: "9f18c840584641dfa9beb98c7dc83e16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f18c840584641dfa9beb98c7dc83e16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38463 } } }
I20260925 12:20:56.759227 19576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16 [sys.catalog]: This master's current role is: LEADER
I20260925 12:20:56.759513 19584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:20:56.760048 19584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:20:56.761663 19584 catalog_manager.cc:1347] Generated new cluster ID: 8b63f94aefcc4193883c4b47e904da48
I20260925 12:20:56.761711 19584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:20:56 dist-test-slave-clmx krb5kdc[19490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:20:56.766994 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:56.756923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52534 (local address 127.0.98.62:38463)
0925 12:20:56.757098 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:56.757105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:56.757125 (+ 20us) server_negotiation.cc:407] Connection header received
0925 12:20:56.757200 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:56.757204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:56.757236 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:56.757356 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:56.758058 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:56.759291 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:56.760367 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:56.760876 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:56.761014 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:56.763630 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:56.763634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:56.763635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:56.763645 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:56.765474 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:56.765841 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:56.765844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:56.765844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:56.765952 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:56.766160 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:56.766160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:56.766161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:56.766411 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:56.766543 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:56.766677 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 12:20:56.766793 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":95,"threads_started":1}
I20260925 12:20:56.774273 19584 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:20:56.774808 19584 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:20:56.780740 19584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16: Generated new TSK 0
I20260925 12:20:56.780907 19584 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:20:56.804203 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38463
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:20:56.880574 19597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:56.880782 19597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:56.880820 19597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:56.881879 19597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:56.881939 19597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:56.881981 19597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:20:56.883383 19597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:56.883630 19597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:56.883852 19597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:56.888303 19605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:56.888301 19602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:56.888336 19603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:56.888602 19597 server_base.cc:1034] running on GCE node
I20260925 12:20:56.888813 19597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:56.889037 19597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:56.890167 19597 hybrid_clock.cc:648] HybridClock initialized: now 1790338856890168 us; error 36 us; skew 500 ppm
Sep 25 12:20:56 dist-test-slave-clmx krb5kdc[19490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:56.892495 19597 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:20:56.892827 19597 webserver.cc:460] Webserver started at http://127.0.98.1:39395/ using document root <none> and password file <none>
I20260925 12:20:56.893008 19597 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:56.893045 19597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:56.893139 19597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:56.894093 19597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "1b71ce5db868408d810a9a1bee3fdf68"
format_stamp: "Formatted at 2026-09-25 12:20:56 on dist-test-slave-clmx"
server_key: "5438d5072796c2553cd1a6ce02ad70fd"
server_key_iv: "433d8e5218855bbe62c59c63c51fc46b"
server_key_version: "encryptionkey@0"
I20260925 12:20:56.894363 19597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1b71ce5db868408d810a9a1bee3fdf68"
format_stamp: "Formatted at 2026-09-25 12:20:56 on dist-test-slave-clmx"
server_key: "5438d5072796c2553cd1a6ce02ad70fd"
server_key_iv: "433d8e5218855bbe62c59c63c51fc46b"
server_key_version: "encryptionkey@0"
I20260925 12:20:56.895443 19597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:20:56.896339 19612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:56.896554 19597 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:20:56.896615 19597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "1b71ce5db868408d810a9a1bee3fdf68"
format_stamp: "Formatted at 2026-09-25 12:20:56 on dist-test-slave-clmx"
server_key: "5438d5072796c2553cd1a6ce02ad70fd"
server_key_iv: "433d8e5218855bbe62c59c63c51fc46b"
server_key_version: "encryptionkey@0"
I20260925 12:20:56.896692 19597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:56.912118 19597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:56.914242 19597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:56.914423 19597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:56.914642 19597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:56.914922 19597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:56.914954 19597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:56.914976 19597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:56.915032 19597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:56.920507 19597 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46607
I20260925 12:20:56.920584 19725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46607 every 8 connection(s)
I20260925 12:20:56.920874 19597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 12:20:56 dist-test-slave-clmx krb5kdc[19490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:20:56.928136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 19597
I20260925 12:20:56.928263 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:20:56.928491 392 external_mini_cluster.cc:1428] Setting key 7e12ff2d0dbce87f16fb8ce428875ad7
I20260925 12:20:56.929960 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:56.921609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44721 (local address 127.0.98.62:38463)
0925 12:20:56.921672 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:56.921674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:56.921875 (+ 201us) server_negotiation.cc:407] Connection header received
0925 12:20:56.922734 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:56.922734 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:56.922747 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:56.922805 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:56.923310 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:56.923895 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:56.924829 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:56.925018 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:56.925067 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:56.926945 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:56.926946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:56.926946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:56.926952 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:56.928508 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:56.928983 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:56.928984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:56.928984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:56.929048 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:56.929343 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:56.929343 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:56.929344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:56.929441 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:56.929504 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:56.929789 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 12:20:56.929870 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 12:20:56.930280 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:56.921588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38463 (local address 127.0.98.1:44721)
0925 12:20:56.921751 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:56.921789 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:56.922652 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:56.922853 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:56.922856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:56.923077 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:56.923243 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:56.923246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:56.923980 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:56.923980 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:56.924761 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:56.924763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:56.925085 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:56.925086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:56.925190 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:56.925607 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:20:56.925615 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:56.926864 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:56.928622 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:56.928629 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:56.928631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:56.928914 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:56.929129 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:56.929130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:56.929130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:56.929275 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:56.929515 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:56.929521 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:56.929666 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 12:20:56.929770 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 12:20:56.929975 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260925 12:20:56.930696 19726 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38463
I20260925 12:20:56.930810 19726 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:56.930974 19726 heartbeater.cc:511] Master 127.0.98.62:38463 requested a full tablet report, sending...
I20260925 12:20:56.931506 19535 ts_manager.cc:194] Registered new tserver with Master: 1b71ce5db868408d810a9a1bee3fdf68 (127.0.98.1:46607)
I20260925 12:20:56.932104 19535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44721
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:20:56.950599 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38463
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:20:57.026171 19731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:57.026355 19731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:57.026377 19731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:57.027447 19731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:57.027483 19731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:57.027506 19731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:20:57.028858 19731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:57.029078 19731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:57.029279 19731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:57.033839 19739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:57.033839 19736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:57.033905 19737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:57.034162 19731 server_base.cc:1034] running on GCE node
I20260925 12:20:57.034379 19731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:57.034647 19731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:57.035832 19731 hybrid_clock.cc:648] HybridClock initialized: now 1790338857035804 us; error 36 us; skew 500 ppm
Sep 25 12:20:57 dist-test-slave-clmx krb5kdc[19490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:57.038163 19731 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:20:57.038539 19731 webserver.cc:460] Webserver started at http://127.0.98.2:33919/ using document root <none> and password file <none>
I20260925 12:20:57.038741 19731 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:57.038801 19731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:57.038938 19731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:57.039888 19731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "59853a9970cb45be993c4e8ca5d9c455"
format_stamp: "Formatted at 2026-09-25 12:20:57 on dist-test-slave-clmx"
server_key: "b69f2cc55a25acd0677eb40a71d89743"
server_key_iv: "494738742dc004884a269c9cf36ae0d5"
server_key_version: "encryptionkey@0"
I20260925 12:20:57.040264 19731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "59853a9970cb45be993c4e8ca5d9c455"
format_stamp: "Formatted at 2026-09-25 12:20:57 on dist-test-slave-clmx"
server_key: "b69f2cc55a25acd0677eb40a71d89743"
server_key_iv: "494738742dc004884a269c9cf36ae0d5"
server_key_version: "encryptionkey@0"
I20260925 12:20:57.041517 19731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:20:57.042148 19746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:57.042328 19731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:20:57.042441 19731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "59853a9970cb45be993c4e8ca5d9c455"
format_stamp: "Formatted at 2026-09-25 12:20:57 on dist-test-slave-clmx"
server_key: "b69f2cc55a25acd0677eb40a71d89743"
server_key_iv: "494738742dc004884a269c9cf36ae0d5"
server_key_version: "encryptionkey@0"
I20260925 12:20:57.042515 19731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:57.055970 19731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:57.057520 19731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:57.057677 19731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:57.057868 19731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:57.058187 19731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:57.058240 19731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:57.058277 19731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:57.058303 19731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:57.064558 19731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39899
I20260925 12:20:57.064590 19859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39899 every 8 connection(s)
I20260925 12:20:57.064913 19731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:20:57 dist-test-slave-clmx krb5kdc[19490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:20:57.072906 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.065590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53389 (local address 127.0.98.62:38463)
0925 12:20:57.065650 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:57.065652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:57.065826 (+ 174us) server_negotiation.cc:407] Connection header received
0925 12:20:57.066654 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:57.066655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:57.066664 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:57.066714 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:57.067260 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.067813 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.068680 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.068847 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.068901 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.070671 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:57.070672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:57.070672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:57.070678 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:57.071902 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:57.072233 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:57.072233 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:57.072234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:57.072279 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:57.072486 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:57.072486 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:57.072486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:57.072570 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:57.072614 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:57.072803 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 12:20:57.072851 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 12:20:57.073050 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.065571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38463 (local address 127.0.98.2:53389)
0925 12:20:57.065714 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:57.065746 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:57.066565 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:57.066762 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:57.066765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:57.066987 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:57.067154 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.067157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.067892 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.067892 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.068607 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.068609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.068903 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.068904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.068985 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.069350 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:20:57.069356 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:57.070593 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:57.071965 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:57.071971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:57.071972 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:57.072155 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:57.072332 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:57.072333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:57.072333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:57.072438 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:57.072618 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:57.072622 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:57.072713 (+ 91us) client_negotiation.cc:769] Sending connection context
0925 12:20:57.072781 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 12:20:57.072877 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":87,"threads_started":1}
I20260925 12:20:57.073529 19860 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38463
I20260925 12:20:57.073607 19860 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:57.073725 19860 heartbeater.cc:511] Master 127.0.98.62:38463 requested a full tablet report, sending...
I20260925 12:20:57.074043 19535 ts_manager.cc:194] Registered new tserver with Master: 59853a9970cb45be993c4e8ca5d9c455 (127.0.98.2:39899)
I20260925 12:20:57.074370 19535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53389
I20260925 12:20:57.074643 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 19731
I20260925 12:20:57.074762 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:20:57.074951 392 external_mini_cluster.cc:1428] Setting key 9cb506ef700f86fa4d549e205bf2bd69
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:20:57.096419 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38463
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:20:57.171912 19865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:57.172096 19865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:57.172129 19865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:57.173249 19865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:57.173305 19865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:57.173345 19865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:20:57.174677 19865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:57.174937 19865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:57.175132 19865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:57.179571 19873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:57.179570 19870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:57.179567 19871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:57.180197 19865 server_base.cc:1034] running on GCE node
I20260925 12:20:57.180382 19865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:57.180548 19865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:57.181701 19865 hybrid_clock.cc:648] HybridClock initialized: now 1790338857181688 us; error 42 us; skew 500 ppm
Sep 25 12:20:57 dist-test-slave-clmx krb5kdc[19490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:57.184069 19865 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:20:57.184404 19865 webserver.cc:460] Webserver started at http://127.0.98.3:41545/ using document root <none> and password file <none>
I20260925 12:20:57.184574 19865 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:57.184612 19865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:57.184693 19865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:57.185580 19865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "2a58c4bdf73a469984c72261e5e1e4db"
format_stamp: "Formatted at 2026-09-25 12:20:57 on dist-test-slave-clmx"
server_key: "db14e212de3fb63e9f9c4ca755688054"
server_key_iv: "fd4477307d25578cb04342ffa0dca87c"
server_key_version: "encryptionkey@0"
I20260925 12:20:57.185854 19865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2a58c4bdf73a469984c72261e5e1e4db"
format_stamp: "Formatted at 2026-09-25 12:20:57 on dist-test-slave-clmx"
server_key: "db14e212de3fb63e9f9c4ca755688054"
server_key_iv: "fd4477307d25578cb04342ffa0dca87c"
server_key_version: "encryptionkey@0"
I20260925 12:20:57.186970 19865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 12:20:57.187565 19880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:57.187776 19865 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:57.187858 19865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "2a58c4bdf73a469984c72261e5e1e4db"
format_stamp: "Formatted at 2026-09-25 12:20:57 on dist-test-slave-clmx"
server_key: "db14e212de3fb63e9f9c4ca755688054"
server_key_iv: "fd4477307d25578cb04342ffa0dca87c"
server_key_version: "encryptionkey@0"
I20260925 12:20:57.187928 19865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:57.203665 19865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:57.205339 19865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:57.205490 19865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:57.205686 19865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:57.205981 19865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:57.206013 19865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:57.206033 19865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:57.206079 19865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:57.211541 19865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33373
I20260925 12:20:57.211637 19993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33373 every 8 connection(s)
I20260925 12:20:57.211910 19865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 12:20:57 dist-test-slave-clmx krb5kdc[19490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:20:57.220217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 19865
I20260925 12:20:57.220325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:20:57.220527 392 external_mini_cluster.cc:1428] Setting key f13ec838f4159c14b5b6668d7f42aa7e
I20260925 12:20:57.220580 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.212600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45283 (local address 127.0.98.62:38463)
0925 12:20:57.212678 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:57.212680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:57.212886 (+ 206us) server_negotiation.cc:407] Connection header received
0925 12:20:57.213725 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:57.213726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:57.213735 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:57.213790 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:57.214318 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.214781 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.215775 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.215957 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.216004 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.217876 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:57.217878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:57.217878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:57.217884 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:57.219172 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:57.219577 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:57.219579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:57.219579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:57.219677 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:57.219915 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:57.219916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:57.219917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:57.220045 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:57.220119 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:57.220416 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 12:20:57.220493 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 12:20:57.220749 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.212571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38463 (local address 127.0.98.3:45283)
0925 12:20:57.212754 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:57.212791 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:57.213640 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:57.213847 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:57.213850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:57.214072 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:57.214238 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.214241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.214857 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.214858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.215639 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.215693 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.216018 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.216019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.216117 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.216490 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:20:57.216496 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:57.217779 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:57.219288 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:57.219293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:57.219295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:57.219503 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:57.219744 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:20:57.219745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:20:57.219746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:20:57.219849 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:20:57.220109 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:57.220115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:57.220246 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 12:20:57.220389 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 12:20:57.220532 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":113,"threads_started":1}
I20260925 12:20:57.221282 19994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38463
I20260925 12:20:57.221369 19994 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:57.221522 19994 heartbeater.cc:511] Master 127.0.98.62:38463 requested a full tablet report, sending...
I20260925 12:20:57.221838 19535 ts_manager.cc:194] Registered new tserver with Master: 2a58c4bdf73a469984c72261e5e1e4db (127.0.98.3:33373)
I20260925 12:20:57.222213 19535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45283
I20260925 12:20:57.232309 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:20:57.239251 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.233576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52538 (local address 127.0.98.62:38463)
0925 12:20:57.233628 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:57.233629 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:57.233724 (+ 95us) server_negotiation.cc:407] Connection header received
0925 12:20:57.233775 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:57.233776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:57.233784 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:57.233832 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:57.234322 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.234752 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.235450 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.235615 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.235703 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.236802 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:57.236803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:57.236804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:57.236810 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:57.238116 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:57.238460 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:57.238461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:57.238461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:57.238510 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:57.238712 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:57.238712 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:57.238713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:57.238882 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:57.238988 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:57.239091 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 12:20:57.239160 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 12:20:57.240640 19535 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:20:57.242129 19535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:20:57.248209 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.244032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33373 (local address 127.0.0.1:40058)
0925 12:20:57.244125 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:57.244157 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:57.244216 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:57.244566 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:57.244570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:57.244572 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:57.244647 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.244649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.245884 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.245885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.247123 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.247125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.247961 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.247961 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.248022 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.248025 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:20:57.248055 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 12:20:57.248117 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":73,"thread_start_us":53,"threads_started":1}
I20260925 12:20:57.248219 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.244064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46607 (local address 127.0.0.1:41858)
0925 12:20:57.244265 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:57.244275 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:57.244344 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:57.244872 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:57.244873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:57.244875 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:57.244915 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.244917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.246143 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.246144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.247002 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.247004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.247920 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.247921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.247995 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.248007 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 12:20:57.248080 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 12:20:57.248117 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":62,"threads_started":1}
I20260925 12:20:57.248459 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.244511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41858 (local address 127.0.98.1:46607)
0925 12:20:57.244708 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:57.244712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:57.244721 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:20:57.244757 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:57.244760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:57.244774 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:57.244986 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:57.245060 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.246049 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.247095 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.247836 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.247915 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.247975 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:57.248100 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 12:20:57.248248 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260925 12:20:57.248591 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.244263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40058 (local address 127.0.98.3:33373)
0925 12:20:57.244421 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:57.244425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:57.244435 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:20:57.244467 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:57.244469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:57.244482 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:57.244564 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:57.244711 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.245782 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.247202 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.247871 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.248398 (+ 527us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.248453 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:57.248476 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 12:20:57.248514 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260925 12:20:57.248740 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.244064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39899 (local address 127.0.0.1:38578)
0925 12:20:57.244416 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:57.244426 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:57.244467 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:57.244738 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:57.244739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:57.244740 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:57.244780 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.244781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.245807 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.245808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.247122 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.247125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.248468 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.248469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.248531 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.248534 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:20:57.248577 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 12:20:57.248617 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":64,"thread_start_us":56,"threads_started":1}
I20260925 12:20:57.249004 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.244288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38578 (local address 127.0.98.2:39899)
0925 12:20:57.244476 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:57.244480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:57.244489 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:20:57.244529 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:57.244531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:57.244547 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:57.244633 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:57.244858 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.245700 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.247684 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.248329 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.248388 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.248438 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:57.248901 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 12:20:57.248940 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":113,"threads_started":1}
I20260925 12:20:57.249178 19660 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2451d7e24048899488cb670abf02a3 (DEFAULT_TABLE table=test-table [id=3b5014a6992643c98d265f00560f7745]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:57.249504 19660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2451d7e24048899488cb670abf02a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:57.249599 19794 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2451d7e24048899488cb670abf02a3 (DEFAULT_TABLE table=test-table [id=3b5014a6992643c98d265f00560f7745]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:57.249935 19794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2451d7e24048899488cb670abf02a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:57.252038 19928 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2451d7e24048899488cb670abf02a3 (DEFAULT_TABLE table=test-table [id=3b5014a6992643c98d265f00560f7745]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:57.252333 19928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2451d7e24048899488cb670abf02a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:57.252279 20008 tablet_bootstrap.cc:492] T 2c2451d7e24048899488cb670abf02a3 P 1b71ce5db868408d810a9a1bee3fdf68: Bootstrap starting.
I20260925 12:20:57.252413 20009 tablet_bootstrap.cc:492] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455: Bootstrap starting.
I20260925 12:20:57.252902 20008 tablet_bootstrap.cc:654] T 2c2451d7e24048899488cb670abf02a3 P 1b71ce5db868408d810a9a1bee3fdf68: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:57.252915 20009 tablet_bootstrap.cc:654] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:57.253453 20008 log.cc:824] T 2c2451d7e24048899488cb670abf02a3 P 1b71ce5db868408d810a9a1bee3fdf68: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:57.253450 20009 log.cc:824] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:57.254101 20009 tablet_bootstrap.cc:492] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455: No bootstrap required, opened a new log
I20260925 12:20:57.254154 20009 ts_tablet_manager.cc:1397] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:20:57.254307 20008 tablet_bootstrap.cc:492] T 2c2451d7e24048899488cb670abf02a3 P 1b71ce5db868408d810a9a1bee3fdf68: No bootstrap required, opened a new log
I20260925 12:20:57.254369 20008 ts_tablet_manager.cc:1397] T 2c2451d7e24048899488cb670abf02a3 P 1b71ce5db868408d810a9a1bee3fdf68: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:20:57.254503 20012 tablet_bootstrap.cc:492] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db: Bootstrap starting.
I20260925 12:20:57.255053 20012 tablet_bootstrap.cc:654] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:57.255579 20012 log.cc:824] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:57.255960 20009 raft_consensus.cc:348] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a58c4bdf73a469984c72261e5e1e4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33373 } } peers { permanent_uuid: "59853a9970cb45be993c4e8ca5d9c455" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39899 } } peers { permanent_uuid: "1b71ce5db868408d810a9a1bee3fdf68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 } }
I20260925 12:20:57.256119 20009 raft_consensus.cc:374] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:57.256040 20008 raft_consensus.cc:348] T 2c2451d7e24048899488cb670abf02a3 P 1b71ce5db868408d810a9a1bee3fdf68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a58c4bdf73a469984c72261e5e1e4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33373 } } peers { permanent_uuid: "59853a9970cb45be993c4e8ca5d9c455" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39899 } } peers { permanent_uuid: "1b71ce5db868408d810a9a1bee3fdf68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 } }
I20260925 12:20:57.256153 20009 raft_consensus.cc:729] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59853a9970cb45be993c4e8ca5d9c455, State: Initialized, Role: FOLLOWER
I20260925 12:20:57.256174 20008 raft_consensus.cc:374] T 2c2451d7e24048899488cb670abf02a3 P 1b71ce5db868408d810a9a1bee3fdf68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:57.256186 20012 tablet_bootstrap.cc:492] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db: No bootstrap required, opened a new log
I20260925 12:20:57.256201 20008 raft_consensus.cc:729] T 2c2451d7e24048899488cb670abf02a3 P 1b71ce5db868408d810a9a1bee3fdf68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b71ce5db868408d810a9a1bee3fdf68, State: Initialized, Role: FOLLOWER
I20260925 12:20:57.256234 20012 ts_tablet_manager.cc:1397] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:20:57.256273 20009 consensus_queue.cc:260] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a58c4bdf73a469984c72261e5e1e4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33373 } } peers { permanent_uuid: "59853a9970cb45be993c4e8ca5d9c455" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39899 } } peers { permanent_uuid: "1b71ce5db868408d810a9a1bee3fdf68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 } }
I20260925 12:20:57.256296 20008 consensus_queue.cc:260] T 2c2451d7e24048899488cb670abf02a3 P 1b71ce5db868408d810a9a1bee3fdf68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a58c4bdf73a469984c72261e5e1e4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33373 } } peers { permanent_uuid: "59853a9970cb45be993c4e8ca5d9c455" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39899 } } peers { permanent_uuid: "1b71ce5db868408d810a9a1bee3fdf68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 } }
I20260925 12:20:57.256521 20008 ts_tablet_manager.cc:1428] T 2c2451d7e24048899488cb670abf02a3 P 1b71ce5db868408d810a9a1bee3fdf68: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:20:57.256523 20009 ts_tablet_manager.cc:1428] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 12:20:57.256685 19860 heartbeater.cc:503] Master 127.0.98.62:38463 was elected leader, sending a full tablet report...
I20260925 12:20:57.256685 19726 heartbeater.cc:503] Master 127.0.98.62:38463 was elected leader, sending a full tablet report...
I20260925 12:20:57.258033 20012 raft_consensus.cc:348] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a58c4bdf73a469984c72261e5e1e4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33373 } } peers { permanent_uuid: "59853a9970cb45be993c4e8ca5d9c455" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39899 } } peers { permanent_uuid: "1b71ce5db868408d810a9a1bee3fdf68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 } }
I20260925 12:20:57.258167 20012 raft_consensus.cc:374] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:57.258210 20012 raft_consensus.cc:729] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a58c4bdf73a469984c72261e5e1e4db, State: Initialized, Role: FOLLOWER
I20260925 12:20:57.258309 20012 consensus_queue.cc:260] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a58c4bdf73a469984c72261e5e1e4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33373 } } peers { permanent_uuid: "59853a9970cb45be993c4e8ca5d9c455" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39899 } } peers { permanent_uuid: "1b71ce5db868408d810a9a1bee3fdf68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 } }
I20260925 12:20:57.258540 20012 ts_tablet_manager.cc:1428] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 12:20:57.258591 19994 heartbeater.cc:503] Master 127.0.98.62:38463 was elected leader, sending a full tablet report...
I20260925 12:20:57.309535 20014 raft_consensus.cc:482] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:20:57.309617 20014 raft_consensus.cc:504] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a58c4bdf73a469984c72261e5e1e4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33373 } } peers { permanent_uuid: "59853a9970cb45be993c4e8ca5d9c455" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39899 } } peers { permanent_uuid: "1b71ce5db868408d810a9a1bee3fdf68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 } }
I20260925 12:20:57.309984 20014 leader_election.cc:288] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a58c4bdf73a469984c72261e5e1e4db (127.0.98.3:33373), 1b71ce5db868408d810a9a1bee3fdf68 (127.0.98.1:46607)
I20260925 12:20:57.312767 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.310145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46607 (local address 127.0.98.2:60469)
0925 12:20:57.310196 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:57.310206 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:57.310244 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:57.310424 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:57.310425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:57.310427 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:57.310494 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.310496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.311157 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.311158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.311986 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.311988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.312537 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.312538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.312620 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.312623 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:20:57.312662 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 12:20:57.312690 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 12:20:57.312800 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.310239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60469 (local address 127.0.98.1:46607)
0925 12:20:57.310284 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:57.310286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:57.310295 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:20:57.310323 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:57.310324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:57.310336 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:57.310412 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:57.310571 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.311082 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.312072 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.312458 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.312537 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.312549 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:57.312694 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 12:20:57.312736 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 12:20:57.312871 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.310234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42893 (local address 127.0.98.3:33373)
0925 12:20:57.310295 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:57.310297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:57.310415 (+ 118us) server_negotiation.cc:407] Connection header received
0925 12:20:57.310466 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:57.310468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:57.310477 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:57.310537 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:57.310712 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.311151 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.312137 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.312468 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.312551 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.312559 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:57.312769 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 12:20:57.312808 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 12:20:57.312901 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.310165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33373 (local address 127.0.98.2:42893)
0925 12:20:57.310335 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:57.310345 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:57.310402 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:57.310571 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:57.310572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:57.310574 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:57.310639 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.310641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.311249 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.311250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.312047 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.312050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.312573 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.312574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.312672 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.312676 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:20:57.312749 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 12:20:57.312787 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
I20260925 12:20:57.313051 19680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2451d7e24048899488cb670abf02a3" candidate_uuid: "59853a9970cb45be993c4e8ca5d9c455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b71ce5db868408d810a9a1bee3fdf68" is_pre_election: true
I20260925 12:20:57.313171 19680 raft_consensus.cc:2454] T 2c2451d7e24048899488cb670abf02a3 P 1b71ce5db868408d810a9a1bee3fdf68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59853a9970cb45be993c4e8ca5d9c455 in term 0.
I20260925 12:20:57.313390 19748 leader_election.cc:302] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b71ce5db868408d810a9a1bee3fdf68, 59853a9970cb45be993c4e8ca5d9c455; no voters:
I20260925 12:20:57.313516 20014 raft_consensus.cc:2781] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:20:57.313566 20014 raft_consensus.cc:482] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:20:57.313593 20014 raft_consensus.cc:3037] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:57.313532 19948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2451d7e24048899488cb670abf02a3" candidate_uuid: "59853a9970cb45be993c4e8ca5d9c455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a58c4bdf73a469984c72261e5e1e4db" is_pre_election: true
I20260925 12:20:57.313661 19948 raft_consensus.cc:2454] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59853a9970cb45be993c4e8ca5d9c455 in term 0.
I20260925 12:20:57.314245 20014 raft_consensus.cc:504] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a58c4bdf73a469984c72261e5e1e4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33373 } } peers { permanent_uuid: "59853a9970cb45be993c4e8ca5d9c455" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39899 } } peers { permanent_uuid: "1b71ce5db868408d810a9a1bee3fdf68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 } }
I20260925 12:20:57.314384 20014 leader_election.cc:288] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [CANDIDATE]: Term 1 election: Requested vote from peers 2a58c4bdf73a469984c72261e5e1e4db (127.0.98.3:33373), 1b71ce5db868408d810a9a1bee3fdf68 (127.0.98.1:46607)
I20260925 12:20:57.314564 19948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2451d7e24048899488cb670abf02a3" candidate_uuid: "59853a9970cb45be993c4e8ca5d9c455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a58c4bdf73a469984c72261e5e1e4db"
I20260925 12:20:57.314558 19680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2451d7e24048899488cb670abf02a3" candidate_uuid: "59853a9970cb45be993c4e8ca5d9c455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b71ce5db868408d810a9a1bee3fdf68"
I20260925 12:20:57.314625 19948 raft_consensus.cc:3037] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:57.314658 19680 raft_consensus.cc:3037] T 2c2451d7e24048899488cb670abf02a3 P 1b71ce5db868408d810a9a1bee3fdf68 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:57.315251 19948 raft_consensus.cc:2454] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59853a9970cb45be993c4e8ca5d9c455 in term 1.
I20260925 12:20:57.315398 19680 raft_consensus.cc:2454] T 2c2451d7e24048899488cb670abf02a3 P 1b71ce5db868408d810a9a1bee3fdf68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59853a9970cb45be993c4e8ca5d9c455 in term 1.
W20260925 12:20:57.315428 19861 tablet.cc:2369] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:20:57.315436 19750 leader_election.cc:302] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a58c4bdf73a469984c72261e5e1e4db, 59853a9970cb45be993c4e8ca5d9c455; no voters:
I20260925 12:20:57.315521 20014 raft_consensus.cc:2781] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:20:57.315659 20014 raft_consensus.cc:686] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [term 1 LEADER]: Becoming Leader. State: Replica: 59853a9970cb45be993c4e8ca5d9c455, State: Running, Role: LEADER
I20260925 12:20:57.315737 20014 consensus_queue.cc:237] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a58c4bdf73a469984c72261e5e1e4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33373 } } peers { permanent_uuid: "59853a9970cb45be993c4e8ca5d9c455" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39899 } } peers { permanent_uuid: "1b71ce5db868408d810a9a1bee3fdf68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 } }
I20260925 12:20:57.316471 19535 catalog_manager.cc:5495] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59853a9970cb45be993c4e8ca5d9c455 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "59853a9970cb45be993c4e8ca5d9c455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a58c4bdf73a469984c72261e5e1e4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59853a9970cb45be993c4e8ca5d9c455" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b71ce5db868408d810a9a1bee3fdf68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 } health_report { overall_health: UNKNOWN } } }
I20260925 12:20:57.338266 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.336015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38580 (local address 127.0.98.2:39899)
0925 12:20:57.336053 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:57.336055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:57.336064 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:20:57.336106 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:57.336106 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:57.336125 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:57.336192 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:20:57.336362 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.336749 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.337528 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.337710 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.337781 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.337960 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:20:57.338038 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:20:57.338127 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:57.338148 (+ 21us) server_negotiation.cc:299] Negotiation successful
0925 12:20:57.338166 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:20:57.339740 19948 raft_consensus.cc:1261] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db [term 1 FOLLOWER]: Refusing update from remote peer 59853a9970cb45be993c4e8ca5d9c455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:20:57.339785 19680 raft_consensus.cc:1261] T 2c2451d7e24048899488cb670abf02a3 P 1b71ce5db868408d810a9a1bee3fdf68 [term 1 FOLLOWER]: Refusing update from remote peer 59853a9970cb45be993c4e8ca5d9c455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:20:57.340070 20014 consensus_queue.cc:1035] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b71ce5db868408d810a9a1bee3fdf68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:20:57.340214 20018 consensus_queue.cc:1035] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a58c4bdf73a469984c72261e5e1e4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:20:57.342296 20021 mvcc.cc:205] Tried to move back new op lower bound from 7333227959661142016 to 7333227959565758464. Current Snapshot: MvccSnapshot[applied={T|T < 7333227959661142016}]
I20260925 12:20:57.342545 20023 mvcc.cc:205] Tried to move back new op lower bound from 7333227959661142016 to 7333227959565758464. Current Snapshot: MvccSnapshot[applied={T|T < 7333227959661142016}]
I20260925 12:20:57.343192 20022 mvcc.cc:205] Tried to move back new op lower bound from 7333227959661142016 to 7333227959565758464. Current Snapshot: MvccSnapshot[applied={T|T < 7333227959661142016}]
I20260925 12:20:57.344537 19535 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:20:57.344627 19535 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:20:57.345346 19535 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f18c840584641dfa9beb98c7dc83e16: Sending DeleteTablet for 3 replicas of tablet 2c2451d7e24048899488cb670abf02a3
I20260925 12:20:57.345655 19660 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c2451d7e24048899488cb670abf02a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:20:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41858
I20260925 12:20:57.345677 19928 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c2451d7e24048899488cb670abf02a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:20:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40058
I20260925 12:20:57.345875 20032 tablet_replica.cc:330] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db: stopping tablet replica
I20260925 12:20:57.345965 20032 raft_consensus.cc:2229] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:20:57.346007 19794 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c2451d7e24048899488cb670abf02a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:20:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38578
I20260925 12:20:57.346060 20032 raft_consensus.cc:2258] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:20:57.346140 20031 tablet_replica.cc:330] T 2c2451d7e24048899488cb670abf02a3 P 1b71ce5db868408d810a9a1bee3fdf68: stopping tablet replica
I20260925 12:20:57.346282 20031 raft_consensus.cc:2229] T 2c2451d7e24048899488cb670abf02a3 P 1b71ce5db868408d810a9a1bee3fdf68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:20:57.346285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 19597
I20260925 12:20:57.346480 20032 ts_tablet_manager.cc:1905] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:20:57.348327 20032 ts_tablet_manager.cc:1918] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:20:57.348397 20032 log.cc:1196] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/2c2451d7e24048899488cb670abf02a3
I20260925 12:20:57.348656 20032 ts_tablet_manager.cc:1939] T 2c2451d7e24048899488cb670abf02a3 P 2a58c4bdf73a469984c72261e5e1e4db: Deleting consensus metadata
I20260925 12:20:57.348892 20033 tablet_replica.cc:330] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455: stopping tablet replica
I20260925 12:20:57.349020 20033 raft_consensus.cc:2229] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:20:57.349023 19522 catalog_manager.cc:4841] TS 2a58c4bdf73a469984c72261e5e1e4db (127.0.98.3:33373): tablet 2c2451d7e24048899488cb670abf02a3 (table test-table [id=3b5014a6992643c98d265f00560f7745]) successfully deleted
I20260925 12:20:57.349164 20033 raft_consensus.cc:2258] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:20:57.349509 20033 ts_tablet_manager.cc:1905] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:20:57.351390 20033 ts_tablet_manager.cc:1918] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:20:57.351455 20033 log.cc:1196] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/2c2451d7e24048899488cb670abf02a3
I20260925 12:20:57.351704 20033 ts_tablet_manager.cc:1939] T 2c2451d7e24048899488cb670abf02a3 P 59853a9970cb45be993c4e8ca5d9c455: Deleting consensus metadata
I20260925 12:20:57.352082 19519 catalog_manager.cc:4841] TS 59853a9970cb45be993c4e8ca5d9c455 (127.0.98.2:39899): tablet 2c2451d7e24048899488cb670abf02a3 (table test-table [id=3b5014a6992643c98d265f00560f7745]) successfully deleted
W20260925 12:20:57.352779 19520 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46607 (error 108)
I20260925 12:20:57.353216 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.353092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46607 (local address 127.0.0.1:41864)
0925 12:20:57.353146 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:57.353184 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 12:20:57.353289 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 19731
W20260925 12:20:57.354031 19520 catalog_manager.cc:4584] TS 1b71ce5db868408d810a9a1bee3fdf68 (127.0.98.1:46607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c2451d7e24048899488cb670abf02a3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46607: connect: Connection refused (error 111)
I20260925 12:20:57.357669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 19865
I20260925 12:20:57.362581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 19503
2026-09-25T12:20:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1817 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T12:20:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:20:57Z Disabled control of system clock
I20260925 12:20:57.388891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39917
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 12:20:57.467712 20038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:57.467998 20038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:57.468034 20038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:57.469681 20038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:20:57.469722 20038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:57.469734 20038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:57.469746 20038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:20:57.469758 20038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:20:57.471387 20038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39917
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:57.471576 20038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:57.471864 20038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:57.476431 20046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:57.476451 20044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:57.476462 20043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:57.476707 20038 server_base.cc:1034] running on GCE node
I20260925 12:20:57.476909 20038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:57.477195 20038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:57.478351 20038 hybrid_clock.cc:648] HybridClock initialized: now 1790338857478345 us; error 36 us; skew 500 ppm
I20260925 12:20:57.479574 20038 webserver.cc:460] Webserver started at http://127.0.98.62:43031/ using document root <none> and password file <none>
I20260925 12:20:57.479851 20038 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:57.479925 20038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:57.480080 20038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:57.481083 20038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "730b7d07d3f4461b9dfc6342bd562c8f"
format_stamp: "Formatted at 2026-09-25 12:20:57 on dist-test-slave-clmx"
server_key: "8b9efe335658fecbf4a272c3e335a26c"
server_key_iv: "6aec4c1eca2ad2eb005f835aee606c6c"
server_key_version: "encryptionkey@0"
I20260925 12:20:57.481465 20038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "730b7d07d3f4461b9dfc6342bd562c8f"
format_stamp: "Formatted at 2026-09-25 12:20:57 on dist-test-slave-clmx"
server_key: "8b9efe335658fecbf4a272c3e335a26c"
server_key_iv: "6aec4c1eca2ad2eb005f835aee606c6c"
server_key_version: "encryptionkey@0"
I20260925 12:20:57.482834 20038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:20:57.483460 20052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:57.483697 20038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:20:57.483791 20038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "730b7d07d3f4461b9dfc6342bd562c8f"
format_stamp: "Formatted at 2026-09-25 12:20:57 on dist-test-slave-clmx"
server_key: "8b9efe335658fecbf4a272c3e335a26c"
server_key_iv: "6aec4c1eca2ad2eb005f835aee606c6c"
server_key_version: "encryptionkey@0"
I20260925 12:20:57.483889 20038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:57.504827 20038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:57.505107 20038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:57.505261 20038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:57.508891 20038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39917
I20260925 12:20:57.508930 20104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39917 every 8 connection(s)
I20260925 12:20:57.509270 20038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:20:57.510176 20105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:57.512369 20105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f: Bootstrap starting.
I20260925 12:20:57.512753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 20038
I20260925 12:20:57.512837 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:20:57.513006 20105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:57.513062 392 external_mini_cluster.cc:1428] Setting key a1b4d4197c72d4e1de8858e9c91f8846
I20260925 12:20:57.513288 20105 log.cc:824] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:57.514002 20105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f: No bootstrap required, opened a new log
I20260925 12:20:57.515988 20105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730b7d07d3f4461b9dfc6342bd562c8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } }
I20260925 12:20:57.516134 20105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:57.516161 20105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730b7d07d3f4461b9dfc6342bd562c8f, State: Initialized, Role: FOLLOWER
I20260925 12:20:57.516261 20105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730b7d07d3f4461b9dfc6342bd562c8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } }
I20260925 12:20:57.516326 20105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:20:57.516352 20105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:20:57.516395 20105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:57.516886 20105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730b7d07d3f4461b9dfc6342bd562c8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } }
I20260925 12:20:57.516997 20105 leader_election.cc:302] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 730b7d07d3f4461b9dfc6342bd562c8f; no voters:
I20260925 12:20:57.517150 20105 leader_election.cc:288] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:20:57.517215 20110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:20:57.517369 20110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f [term 1 LEADER]: Becoming Leader. State: Replica: 730b7d07d3f4461b9dfc6342bd562c8f, State: Running, Role: LEADER
I20260925 12:20:57.517407 20105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:20:57.517486 20110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730b7d07d3f4461b9dfc6342bd562c8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } }
I20260925 12:20:57.517838 20111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "730b7d07d3f4461b9dfc6342bd562c8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730b7d07d3f4461b9dfc6342bd562c8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } }
I20260925 12:20:57.517988 20111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f [sys.catalog]: This master's current role is: LEADER
I20260925 12:20:57.517825 20110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 730b7d07d3f4461b9dfc6342bd562c8f. Latest consensus state: current_term: 1 leader_uuid: "730b7d07d3f4461b9dfc6342bd562c8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730b7d07d3f4461b9dfc6342bd562c8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } }
I20260925 12:20:57.518277 20110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f [sys.catalog]: This master's current role is: LEADER
I20260925 12:20:57.518445 20119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:20:57.518987 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.513556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.0.98.62:39917)
0925 12:20:57.513758 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:57.513765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:57.513789 (+ 24us) server_negotiation.cc:407] Connection header received
0925 12:20:57.514229 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:57.514236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:57.514289 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:57.514397 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:57.514632 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.515425 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.516514 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.516904 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.517047 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.517713 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:57.517716 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:57.517717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:20:57.517920 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:57.517960 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:20:57.517974 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:57.518290 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:57.518322 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 12:20:57.518403 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260925 12:20:57.519387 20119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:20:57.520979 20119 catalog_manager.cc:1347] Generated new cluster ID: 2b311f28fdc245df8421e247eb385ba0
I20260925 12:20:57.521039 20119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:20:57.539364 20119 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:20:57.539947 20119 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:20:57.542932 20119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f: Generated new TSK 0
I20260925 12:20:57.543066 20119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:20:57.545836 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39917
--builtin_ntp_servers=127.0.98.20:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 12:20:57.623490 20129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:57.623744 20129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:57.623780 20129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:57.624917 20129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:57.624979 20129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:57.625022 20129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:20:57.626339 20129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:57.626571 20129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:57.626834 20129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:57.632143 20137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:57.632138 20135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:57.632365 20129 server_base.cc:1034] running on GCE node
W20260925 12:20:57.632130 20134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:57.632613 20129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:57.632858 20129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:57.634024 20129 hybrid_clock.cc:648] HybridClock initialized: now 1790338857634005 us; error 37 us; skew 500 ppm
I20260925 12:20:57.635293 20129 webserver.cc:460] Webserver started at http://127.0.98.1:40389/ using document root <none> and password file <none>
I20260925 12:20:57.635558 20129 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:57.635641 20129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:57.635769 20129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:57.637058 20129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-0/data/instance:
uuid: "011cc2d6eae842cb8b4faea2a687125f"
format_stamp: "Formatted at 2026-09-25 12:20:57 on dist-test-slave-clmx"
server_key: "041c855b2d8eec2e2164f4d733462734"
server_key_iv: "e035debbf92fd4887cfe0db58ed7fb71"
server_key_version: "encryptionkey@0"
I20260925 12:20:57.637337 20129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-0/wal/instance:
uuid: "011cc2d6eae842cb8b4faea2a687125f"
format_stamp: "Formatted at 2026-09-25 12:20:57 on dist-test-slave-clmx"
server_key: "041c855b2d8eec2e2164f4d733462734"
server_key_iv: "e035debbf92fd4887cfe0db58ed7fb71"
server_key_version: "encryptionkey@0"
I20260925 12:20:57.638465 20129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:20:57.639088 20143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:57.639283 20129 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:20:57.639345 20129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-0/wal
uuid: "011cc2d6eae842cb8b4faea2a687125f"
format_stamp: "Formatted at 2026-09-25 12:20:57 on dist-test-slave-clmx"
server_key: "041c855b2d8eec2e2164f4d733462734"
server_key_iv: "e035debbf92fd4887cfe0db58ed7fb71"
server_key_version: "encryptionkey@0"
I20260925 12:20:57.639432 20129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:57.650422 20129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:57.650681 20129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:57.650828 20129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:57.651041 20129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:57.651358 20129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:57.651391 20129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:57.651410 20129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:57.651453 20129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:57.656833 20129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33119
I20260925 12:20:57.656904 20256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33119 every 8 connection(s)
I20260925 12:20:57.657195 20129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:20:57.660369 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 20129
I20260925 12:20:57.660480 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:20:57.660655 392 external_mini_cluster.cc:1428] Setting key 2e36af7107a4c6040b4edefd196c0d1e
I20260925 12:20:57.661705 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39917
--builtin_ntp_servers=127.0.98.20:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 12:20:57.662379 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.657860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44077 (local address 127.0.98.62:39917)
0925 12:20:57.657903 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:57.657905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:57.658121 (+ 216us) server_negotiation.cc:407] Connection header received
0925 12:20:57.658563 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:57.658564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:57.658572 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:57.658624 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:57.659196 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.659714 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.660745 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.661025 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.661095 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.662049 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:57.662050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:57.662051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:20:57.662055 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:57.662068 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:20:57.662072 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:57.662170 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:57.662251 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 12:20:57.662275 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:20:57.662547 20259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.657820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39917 (local address 127.0.98.1:44077)
0925 12:20:57.657998 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:57.658036 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:57.658471 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:57.658671 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:57.658674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:57.658681 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:57.659129 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.659134 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.659808 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.659809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.660659 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.660661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.661117 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.661118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.661294 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.661875 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:20:57.661876 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:57.661914 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:20:57.661956 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:57.662174 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:57.662180 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:57.662182 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:20:57.662247 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 12:20:57.662344 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260925 12:20:57.663033 20257 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39917
I20260925 12:20:57.663122 20257 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:57.663331 20257 heartbeater.cc:511] Master 127.0.98.62:39917 requested a full tablet report, sending...
I20260925 12:20:57.663826 20069 ts_manager.cc:194] Registered new tserver with Master: 011cc2d6eae842cb8b4faea2a687125f (127.0.98.1:33119)
I20260925 12:20:57.664434 20069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:44077
W20260925 12:20:57.737831 20260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:57.738052 20260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:57.738101 20260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:57.739153 20260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:57.739212 20260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:57.739266 20260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:20:57.740646 20260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:57.740895 20260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:57.741153 20260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:57.745858 20265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:57.745869 20266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:57.745909 20268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:57.746111 20260 server_base.cc:1034] running on GCE node
I20260925 12:20:57.746320 20260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:57.746526 20260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:57.747671 20260 hybrid_clock.cc:648] HybridClock initialized: now 1790338857747666 us; error 32 us; skew 500 ppm
I20260925 12:20:57.748931 20260 webserver.cc:460] Webserver started at http://127.0.98.2:38077/ using document root <none> and password file <none>
I20260925 12:20:57.749153 20260 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:57.749235 20260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:57.749354 20260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:57.750322 20260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-1/data/instance:
uuid: "14cdf173d0bd4f109a840f4b095fdabe"
format_stamp: "Formatted at 2026-09-25 12:20:57 on dist-test-slave-clmx"
server_key: "a3d08c45ea73fc2b593c7376020ada14"
server_key_iv: "cf457756eb5ce01b6bada91cc31c86ba"
server_key_version: "encryptionkey@0"
I20260925 12:20:57.750663 20260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-1/wal/instance:
uuid: "14cdf173d0bd4f109a840f4b095fdabe"
format_stamp: "Formatted at 2026-09-25 12:20:57 on dist-test-slave-clmx"
server_key: "a3d08c45ea73fc2b593c7376020ada14"
server_key_iv: "cf457756eb5ce01b6bada91cc31c86ba"
server_key_version: "encryptionkey@0"
I20260925 12:20:57.751878 20260 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:20:57.752470 20274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:57.752607 20260 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:20:57.752691 20260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-1/wal
uuid: "14cdf173d0bd4f109a840f4b095fdabe"
format_stamp: "Formatted at 2026-09-25 12:20:57 on dist-test-slave-clmx"
server_key: "a3d08c45ea73fc2b593c7376020ada14"
server_key_iv: "cf457756eb5ce01b6bada91cc31c86ba"
server_key_version: "encryptionkey@0"
I20260925 12:20:57.752764 20260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:57.778476 20260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:57.778733 20260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:57.778870 20260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:57.779066 20260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:57.779356 20260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:57.779404 20260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:57.779436 20260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:57.779460 20260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:57.784968 20260 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33907
I20260925 12:20:57.785010 20387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33907 every 8 connection(s)
I20260925 12:20:57.785306 20260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:20:57.786613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 20260
I20260925 12:20:57.786753 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:20:57.786963 392 external_mini_cluster.cc:1428] Setting key 89faa66fc059d6017316595c2820f03e
I20260925 12:20:57.787940 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39917
--builtin_ntp_servers=127.0.98.20:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 12:20:57.791069 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.786004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49151 (local address 127.0.98.62:39917)
0925 12:20:57.786075 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:57.786077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:57.786272 (+ 195us) server_negotiation.cc:407] Connection header received
0925 12:20:57.786890 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:57.786891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:57.786900 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:57.786968 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:57.787764 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.788319 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.789607 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.789890 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.789967 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.790785 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:57.790787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:57.790787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:20:57.790792 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:57.790806 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:20:57.790811 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:57.790881 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:57.790932 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 12:20:57.790954 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 12:20:57.791214 20390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.785995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39917 (local address 127.0.98.2:49151)
0925 12:20:57.786157 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:57.786194 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:57.786784 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:57.787041 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:57.787045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:57.787055 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:57.787658 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.787664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.788404 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.788405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.789517 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.789519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.789971 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.789971 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.790088 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.790653 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:20:57.790654 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:57.790680 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:20:57.790700 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:57.790882 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:57.790884 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:57.790886 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:20:57.790930 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 12:20:57.790999 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260925 12:20:57.791730 20388 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39917
I20260925 12:20:57.791837 20388 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:57.792032 20388 heartbeater.cc:511] Master 127.0.98.62:39917 requested a full tablet report, sending...
I20260925 12:20:57.792382 20069 ts_manager.cc:194] Registered new tserver with Master: 14cdf173d0bd4f109a840f4b095fdabe (127.0.98.2:33907)
I20260925 12:20:57.792724 20069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49151
W20260925 12:20:57.863754 20391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:57.863921 20391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:57.863942 20391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:57.865012 20391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:57.865049 20391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:57.865098 20391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:20:57.866269 20391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:57.866477 20391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:57.866748 20391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:57.871127 20396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:57.871096 20397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:57.871116 20399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:57.871444 20391 server_base.cc:1034] running on GCE node
I20260925 12:20:57.871709 20391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:57.871935 20391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:57.873075 20391 hybrid_clock.cc:648] HybridClock initialized: now 1790338857873063 us; error 29 us; skew 500 ppm
I20260925 12:20:57.874158 20391 webserver.cc:460] Webserver started at http://127.0.98.3:35391/ using document root <none> and password file <none>
I20260925 12:20:57.874408 20391 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:57.874450 20391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:57.874607 20391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:57.875554 20391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-2/data/instance:
uuid: "b6504e1a7e114f4e9bcc6238b2751dde"
format_stamp: "Formatted at 2026-09-25 12:20:57 on dist-test-slave-clmx"
server_key: "3614d0bafba2020c5b64801f6890a088"
server_key_iv: "a7e3ccf75b2b04045d5a7ec4f3729df1"
server_key_version: "encryptionkey@0"
I20260925 12:20:57.875916 20391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b6504e1a7e114f4e9bcc6238b2751dde"
format_stamp: "Formatted at 2026-09-25 12:20:57 on dist-test-slave-clmx"
server_key: "3614d0bafba2020c5b64801f6890a088"
server_key_iv: "a7e3ccf75b2b04045d5a7ec4f3729df1"
server_key_version: "encryptionkey@0"
I20260925 12:20:57.876999 20391 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:20:57.877591 20405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:57.877760 20391 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:57.877844 20391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-2/wal
uuid: "b6504e1a7e114f4e9bcc6238b2751dde"
format_stamp: "Formatted at 2026-09-25 12:20:57 on dist-test-slave-clmx"
server_key: "3614d0bafba2020c5b64801f6890a088"
server_key_iv: "a7e3ccf75b2b04045d5a7ec4f3729df1"
server_key_version: "encryptionkey@0"
I20260925 12:20:57.877916 20391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:57.889153 20391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:57.889397 20391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:57.889530 20391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:57.889719 20391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:20:57.890003 20391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:20:57.890055 20391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:57.890107 20391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:20:57.890144 20391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:57.895820 20391 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39631
I20260925 12:20:57.895875 20518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39631 every 8 connection(s)
I20260925 12:20:57.896127 20391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:20:57.900573 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.896693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35113 (local address 127.0.98.62:39917)
0925 12:20:57.896732 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:57.896734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:57.896919 (+ 185us) server_negotiation.cc:407] Connection header received
0925 12:20:57.897350 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:57.897351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:57.897363 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:57.897407 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:57.897959 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.898416 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.899339 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.899518 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.899589 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.900288 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:57.900289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:57.900290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:20:57.900294 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:57.900307 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:20:57.900311 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:57.900383 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:57.900473 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 12:20:57.900497 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 12:20:57.900770 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.896669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39917 (local address 127.0.98.3:35113)
0925 12:20:57.896815 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:57.896847 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:57.897258 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:57.897459 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:57.897462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:57.897470 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:20:57.897877 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.897881 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.898504 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.898505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.899239 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.899240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.899595 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.899596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.899732 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.900147 (+ 415us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:20:57.900147 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:20:57.900171 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:20:57.900190 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:20:57.900383 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:20:57.900388 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:20:57.900391 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:20:57.900475 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 12:20:57.900566 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":60,"threads_started":1}
I20260925 12:20:57.901190 20519 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39917
I20260925 12:20:57.901264 20519 heartbeater.cc:464] Registering TS with master...
I20260925 12:20:57.901463 20519 heartbeater.cc:511] Master 127.0.98.62:39917 requested a full tablet report, sending...
I20260925 12:20:57.901775 20069 ts_manager.cc:194] Registered new tserver with Master: b6504e1a7e114f4e9bcc6238b2751dde (127.0.98.3:39631)
I20260925 12:20:57.902122 20069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:35113
I20260925 12:20:57.902805 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 20391
I20260925 12:20:57.902886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:20:57.903005 392 external_mini_cluster.cc:1428] Setting key 1c3efa90d1882826714eaa3542ba8aa2
I20260925 12:20:57.903800 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:20:57.907711 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.905085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43856 (local address 127.0.98.62:39917)
0925 12:20:57.905150 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:57.905152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:57.905223 (+ 71us) server_negotiation.cc:407] Connection header received
0925 12:20:57.905296 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:57.905297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:57.905305 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:57.905358 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:57.905529 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.905977 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.906714 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.906894 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.906965 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.907482 (+ 517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:57.907483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:57.907484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:20:57.907488 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:57.907498 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:20:57.907502 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:57.907554 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:57.907601 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 12:20:57.907637 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 12:20:57.909004 20069 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:20:57.910516 20069 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:20:57.916280 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.912463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33907 (local address 127.0.0.1:59724)
0925 12:20:57.912617 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:57.912652 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:57.912728 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:57.912965 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:57.912967 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:57.912969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:57.913022 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.913024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.914079 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.914080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.915073 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.915075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.916012 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.916013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.916081 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.916093 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 12:20:57.916169 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 12:20:57.916212 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":6,"thread_start_us":66,"threads_started":1}
I20260925 12:20:57.916296 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.912622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59724 (local address 127.0.98.2:33907)
0925 12:20:57.912801 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:57.912805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:57.912813 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:20:57.912848 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:57.912849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:57.912881 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:57.912960 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:57.913077 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.913970 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.915164 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.915936 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.916002 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.916081 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:57.916182 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 12:20:57.916218 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":80,"threads_started":1}
I20260925 12:20:57.916810 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.912467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33119 (local address 127.0.0.1:44386)
0925 12:20:57.912994 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:57.913003 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:57.913066 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:57.913302 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:57.913302 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:57.913304 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:57.913354 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.913355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.914655 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.914656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.915676 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.915679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.916573 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.916574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.916631 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.916634 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:20:57.916689 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 12:20:57.916752 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":99,"threads_started":1}
I20260925 12:20:57.916849 20532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.912793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.0.98.1:33119)
0925 12:20:57.912963 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:57.912968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:57.913067 (+ 99us) server_negotiation.cc:407] Connection header received
0925 12:20:57.913143 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:57.913145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:57.913162 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:57.913237 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:57.913429 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.914565 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.915755 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.916403 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.916461 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.916525 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:57.916722 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 12:20:57.916762 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":108,"threads_started":1}
I20260925 12:20:57.916996 20322 tablet_service.cc:1467] Processing CreateTablet for tablet f63265c917d642ebbd861c17ea08b69e (DEFAULT_TABLE table=test-table [id=b44c57457d4e4fc5b16e2bc993dde548]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:57.917383 20322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63265c917d642ebbd861c17ea08b69e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:57.917500 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.912472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39631 (local address 127.0.0.1:59954)
0925 12:20:57.913154 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:57.913162 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:57.913208 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:57.913428 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:57.913429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:57.913431 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:57.913472 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.913474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.914686 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.914687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.915714 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:57.915717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.917268 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.917269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:57.917345 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.917348 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:20:57.917396 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 12:20:57.917433 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":74,"threads_started":1}
I20260925 12:20:57.917526 20191 tablet_service.cc:1467] Processing CreateTablet for tablet f63265c917d642ebbd861c17ea08b69e (DEFAULT_TABLE table=test-table [id=b44c57457d4e4fc5b16e2bc993dde548]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:57.917718 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:57.912656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59954 (local address 127.0.98.3:39631)
0925 12:20:57.912790 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:57.912794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:57.913235 (+ 441us) server_negotiation.cc:407] Connection header received
0925 12:20:57.913276 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:57.913279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:57.913293 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:57.913377 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:57.913557 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.914604 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.915811 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:57.916795 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:57.917094 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:57.917187 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:57.917410 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 12:20:57.917492 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260925 12:20:57.917848 20191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63265c917d642ebbd861c17ea08b69e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:57.920625 20533 tablet_bootstrap.cc:492] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe: Bootstrap starting.
I20260925 12:20:57.921021 20534 tablet_bootstrap.cc:492] T f63265c917d642ebbd861c17ea08b69e P 011cc2d6eae842cb8b4faea2a687125f: Bootstrap starting.
I20260925 12:20:57.920980 20453 tablet_service.cc:1467] Processing CreateTablet for tablet f63265c917d642ebbd861c17ea08b69e (DEFAULT_TABLE table=test-table [id=b44c57457d4e4fc5b16e2bc993dde548]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:20:57.921299 20453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63265c917d642ebbd861c17ea08b69e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:57.921451 20533 tablet_bootstrap.cc:654] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:57.921653 20534 tablet_bootstrap.cc:654] T f63265c917d642ebbd861c17ea08b69e P 011cc2d6eae842cb8b4faea2a687125f: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:57.921761 20533 log.cc:824] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:57.921947 20534 log.cc:824] T f63265c917d642ebbd861c17ea08b69e P 011cc2d6eae842cb8b4faea2a687125f: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:57.922570 20533 tablet_bootstrap.cc:492] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe: No bootstrap required, opened a new log
I20260925 12:20:57.922642 20533 ts_tablet_manager.cc:1397] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:20:57.922785 20534 tablet_bootstrap.cc:492] T f63265c917d642ebbd861c17ea08b69e P 011cc2d6eae842cb8b4faea2a687125f: No bootstrap required, opened a new log
I20260925 12:20:57.922891 20534 ts_tablet_manager.cc:1397] T f63265c917d642ebbd861c17ea08b69e P 011cc2d6eae842cb8b4faea2a687125f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:20:57.924477 20533 raft_consensus.cc:348] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cdf173d0bd4f109a840f4b095fdabe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33907 } } peers { permanent_uuid: "011cc2d6eae842cb8b4faea2a687125f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33119 } } peers { permanent_uuid: "b6504e1a7e114f4e9bcc6238b2751dde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39631 } }
I20260925 12:20:57.924643 20537 tablet_bootstrap.cc:492] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde: Bootstrap starting.
I20260925 12:20:57.924661 20533 raft_consensus.cc:374] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:57.924701 20533 raft_consensus.cc:729] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14cdf173d0bd4f109a840f4b095fdabe, State: Initialized, Role: FOLLOWER
I20260925 12:20:57.924804 20533 consensus_queue.cc:260] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cdf173d0bd4f109a840f4b095fdabe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33907 } } peers { permanent_uuid: "011cc2d6eae842cb8b4faea2a687125f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33119 } } peers { permanent_uuid: "b6504e1a7e114f4e9bcc6238b2751dde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39631 } }
I20260925 12:20:57.925065 20533 ts_tablet_manager.cc:1428] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 12:20:57.925177 20388 heartbeater.cc:503] Master 127.0.98.62:39917 was elected leader, sending a full tablet report...
I20260925 12:20:57.925520 20537 tablet_bootstrap.cc:654] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:57.925674 20534 raft_consensus.cc:348] T f63265c917d642ebbd861c17ea08b69e P 011cc2d6eae842cb8b4faea2a687125f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cdf173d0bd4f109a840f4b095fdabe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33907 } } peers { permanent_uuid: "011cc2d6eae842cb8b4faea2a687125f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33119 } } peers { permanent_uuid: "b6504e1a7e114f4e9bcc6238b2751dde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39631 } }
I20260925 12:20:57.925805 20534 raft_consensus.cc:374] T f63265c917d642ebbd861c17ea08b69e P 011cc2d6eae842cb8b4faea2a687125f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:57.925843 20534 raft_consensus.cc:729] T f63265c917d642ebbd861c17ea08b69e P 011cc2d6eae842cb8b4faea2a687125f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011cc2d6eae842cb8b4faea2a687125f, State: Initialized, Role: FOLLOWER
I20260925 12:20:57.925892 20537 log.cc:824] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:57.925967 20534 consensus_queue.cc:260] T f63265c917d642ebbd861c17ea08b69e P 011cc2d6eae842cb8b4faea2a687125f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cdf173d0bd4f109a840f4b095fdabe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33907 } } peers { permanent_uuid: "011cc2d6eae842cb8b4faea2a687125f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33119 } } peers { permanent_uuid: "b6504e1a7e114f4e9bcc6238b2751dde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39631 } }
I20260925 12:20:57.926216 20534 ts_tablet_manager.cc:1428] T f63265c917d642ebbd861c17ea08b69e P 011cc2d6eae842cb8b4faea2a687125f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:20:57.926250 20257 heartbeater.cc:503] Master 127.0.98.62:39917 was elected leader, sending a full tablet report...
I20260925 12:20:57.926951 20537 tablet_bootstrap.cc:492] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde: No bootstrap required, opened a new log
I20260925 12:20:57.927012 20537 ts_tablet_manager.cc:1397] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:20:57.928728 20537 raft_consensus.cc:348] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cdf173d0bd4f109a840f4b095fdabe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33907 } } peers { permanent_uuid: "011cc2d6eae842cb8b4faea2a687125f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33119 } } peers { permanent_uuid: "b6504e1a7e114f4e9bcc6238b2751dde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39631 } }
I20260925 12:20:57.928843 20537 raft_consensus.cc:374] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:57.928865 20537 raft_consensus.cc:729] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6504e1a7e114f4e9bcc6238b2751dde, State: Initialized, Role: FOLLOWER
I20260925 12:20:57.928960 20537 consensus_queue.cc:260] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cdf173d0bd4f109a840f4b095fdabe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33907 } } peers { permanent_uuid: "011cc2d6eae842cb8b4faea2a687125f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33119 } } peers { permanent_uuid: "b6504e1a7e114f4e9bcc6238b2751dde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39631 } }
I20260925 12:20:57.929152 20537 ts_tablet_manager.cc:1428] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:20:57.929230 20519 heartbeater.cc:503] Master 127.0.98.62:39917 was elected leader, sending a full tablet report...
W20260925 12:20:58.035851 20389 tablet.cc:2369] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:20:58.125669 20541 raft_consensus.cc:482] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:20:58.125778 20541 raft_consensus.cc:504] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cdf173d0bd4f109a840f4b095fdabe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33907 } } peers { permanent_uuid: "011cc2d6eae842cb8b4faea2a687125f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33119 } } peers { permanent_uuid: "b6504e1a7e114f4e9bcc6238b2751dde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39631 } }
I20260925 12:20:58.126157 20541 leader_election.cc:288] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14cdf173d0bd4f109a840f4b095fdabe (127.0.98.2:33907), 011cc2d6eae842cb8b4faea2a687125f (127.0.98.1:33119)
I20260925 12:20:58.128818 20532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:58.126446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38727 (local address 127.0.98.1:33119)
0925 12:20:58.126498 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:58.126500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:58.126518 (+ 18us) server_negotiation.cc:407] Connection header received
0925 12:20:58.126550 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:58.126550 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:58.126557 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:58.126618 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:58.126762 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:58.127225 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:58.128148 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:58.128484 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:58.128581 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:58.128589 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:58.128710 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 12:20:58.128740 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 12:20:58.128818 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:58.126341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33119 (local address 127.0.98.3:38727)
0925 12:20:58.126378 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:58.126389 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:58.126444 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:58.126621 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:58.126621 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:58.126623 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:58.126696 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:58.126697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:58.127333 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:58.127334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:58.128064 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:58.128066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:58.128583 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:58.128584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:58.128647 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:58.128649 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:20:58.128705 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 12:20:58.128740 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 12:20:58.128981 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:58.126432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44047 (local address 127.0.98.2:33907)
0925 12:20:58.126488 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:58.126490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:58.126636 (+ 146us) server_negotiation.cc:407] Connection header received
0925 12:20:58.126703 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:58.126704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:58.126713 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:58.126780 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:20:58.126932 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:58.127381 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:58.128274 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:58.128607 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:58.128675 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:58.128683 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:58.128858 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 12:20:58.128898 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 12:20:58.129092 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:58.126341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33907 (local address 127.0.98.3:44047)
0925 12:20:58.126565 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:58.126578 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:20:58.126650 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:20:58.126784 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:20:58.126784 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:20:58.126786 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:20:58.126854 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:58.126860 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:58.127457 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:58.127458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:58.128180 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:20:58.128182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:20:58.128687 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:20:58.128688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:20:58.128766 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:58.128768 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:20:58.128849 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 12:20:58.129009 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":72,"thread_start_us":98,"threads_started":1}
I20260925 12:20:58.129146 20211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63265c917d642ebbd861c17ea08b69e" candidate_uuid: "b6504e1a7e114f4e9bcc6238b2751dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011cc2d6eae842cb8b4faea2a687125f" is_pre_election: true
I20260925 12:20:58.129298 20211 raft_consensus.cc:2454] T f63265c917d642ebbd861c17ea08b69e P 011cc2d6eae842cb8b4faea2a687125f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6504e1a7e114f4e9bcc6238b2751dde in term 0.
I20260925 12:20:58.129372 20342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63265c917d642ebbd861c17ea08b69e" candidate_uuid: "b6504e1a7e114f4e9bcc6238b2751dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14cdf173d0bd4f109a840f4b095fdabe" is_pre_election: true
I20260925 12:20:58.129528 20342 raft_consensus.cc:2454] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6504e1a7e114f4e9bcc6238b2751dde in term 0.
I20260925 12:20:58.129534 20409 leader_election.cc:302] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 011cc2d6eae842cb8b4faea2a687125f, b6504e1a7e114f4e9bcc6238b2751dde; no voters:
I20260925 12:20:58.129624 20541 raft_consensus.cc:2781] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:20:58.129709 20541 raft_consensus.cc:482] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:20:58.129733 20541 raft_consensus.cc:3037] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:58.130386 20541 raft_consensus.cc:504] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cdf173d0bd4f109a840f4b095fdabe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33907 } } peers { permanent_uuid: "011cc2d6eae842cb8b4faea2a687125f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33119 } } peers { permanent_uuid: "b6504e1a7e114f4e9bcc6238b2751dde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39631 } }
I20260925 12:20:58.130510 20541 leader_election.cc:288] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [CANDIDATE]: Term 1 election: Requested vote from peers 14cdf173d0bd4f109a840f4b095fdabe (127.0.98.2:33907), 011cc2d6eae842cb8b4faea2a687125f (127.0.98.1:33119)
I20260925 12:20:58.130652 20342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63265c917d642ebbd861c17ea08b69e" candidate_uuid: "b6504e1a7e114f4e9bcc6238b2751dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14cdf173d0bd4f109a840f4b095fdabe"
I20260925 12:20:58.130666 20211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63265c917d642ebbd861c17ea08b69e" candidate_uuid: "b6504e1a7e114f4e9bcc6238b2751dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011cc2d6eae842cb8b4faea2a687125f"
I20260925 12:20:58.130730 20342 raft_consensus.cc:3037] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:58.130730 20211 raft_consensus.cc:3037] T f63265c917d642ebbd861c17ea08b69e P 011cc2d6eae842cb8b4faea2a687125f [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:58.131588 20211 raft_consensus.cc:2454] T f63265c917d642ebbd861c17ea08b69e P 011cc2d6eae842cb8b4faea2a687125f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6504e1a7e114f4e9bcc6238b2751dde in term 1.
I20260925 12:20:58.131588 20342 raft_consensus.cc:2454] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6504e1a7e114f4e9bcc6238b2751dde in term 1.
I20260925 12:20:58.131762 20408 leader_election.cc:302] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14cdf173d0bd4f109a840f4b095fdabe, b6504e1a7e114f4e9bcc6238b2751dde; no voters:
I20260925 12:20:58.131846 20541 raft_consensus.cc:2781] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:20:58.131966 20541 raft_consensus.cc:686] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [term 1 LEADER]: Becoming Leader. State: Replica: b6504e1a7e114f4e9bcc6238b2751dde, State: Running, Role: LEADER
I20260925 12:20:58.132048 20541 consensus_queue.cc:237] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cdf173d0bd4f109a840f4b095fdabe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33907 } } peers { permanent_uuid: "011cc2d6eae842cb8b4faea2a687125f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33119 } } peers { permanent_uuid: "b6504e1a7e114f4e9bcc6238b2751dde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39631 } }
I20260925 12:20:58.132845 20068 catalog_manager.cc:5495] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde reported cstate change: term changed from 0 to 1, leader changed from <none> to b6504e1a7e114f4e9bcc6238b2751dde (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b6504e1a7e114f4e9bcc6238b2751dde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cdf173d0bd4f109a840f4b095fdabe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "011cc2d6eae842cb8b4faea2a687125f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6504e1a7e114f4e9bcc6238b2751dde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39631 } health_report { overall_health: HEALTHY } } }
I20260925 12:20:58.142903 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:58.140776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59960 (local address 127.0.98.3:39631)
0925 12:20:58.140819 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:58.140821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:58.140828 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:20:58.140905 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:58.140906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:58.140918 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:58.140972 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:20:58.141069 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:58.141475 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:58.142226 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:58.142392 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:58.142441 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:58.142551 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:20:58.142661 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:20:58.142744 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:58.142830 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 12:20:58.142845 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:20:58.144747 20342 raft_consensus.cc:1261] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe [term 1 FOLLOWER]: Refusing update from remote peer b6504e1a7e114f4e9bcc6238b2751dde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:20:58.144747 20211 raft_consensus.cc:1261] T f63265c917d642ebbd861c17ea08b69e P 011cc2d6eae842cb8b4faea2a687125f [term 1 FOLLOWER]: Refusing update from remote peer b6504e1a7e114f4e9bcc6238b2751dde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:20:58.145063 20543 consensus_queue.cc:1035] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "011cc2d6eae842cb8b4faea2a687125f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:20:58.145185 20541 consensus_queue.cc:1035] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "14cdf173d0bd4f109a840f4b095fdabe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 12:20:58.146570 20520 tablet.cc:2369] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:20:58.147805 20548 mvcc.cc:205] Tried to move back new op lower bound from 7333227962958434304 to 7333227962909487104. Current Snapshot: MvccSnapshot[applied={T|T < 7333227962958434304}]
I20260925 12:20:58.149300 20069 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:20:58.149401 20069 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:20:58.150039 20069 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 730b7d07d3f4461b9dfc6342bd562c8f: Sending DeleteTablet for 3 replicas of tablet f63265c917d642ebbd861c17ea08b69e
I20260925 12:20:58.150377 20322 tablet_service.cc:1514] Processing DeleteTablet for tablet f63265c917d642ebbd861c17ea08b69e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:20:58 UTC) from {username='slave'} at 127.0.0.1:59724
I20260925 12:20:58.150466 20453 tablet_service.cc:1514] Processing DeleteTablet for tablet f63265c917d642ebbd861c17ea08b69e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:20:58 UTC) from {username='slave'} at 127.0.0.1:59954
I20260925 12:20:58.150599 20191 tablet_service.cc:1514] Processing DeleteTablet for tablet f63265c917d642ebbd861c17ea08b69e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:20:58 UTC) from {username='slave'} at 127.0.0.1:44386
I20260925 12:20:58.150635 20556 tablet_replica.cc:330] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe: stopping tablet replica
I20260925 12:20:58.150691 20557 tablet_replica.cc:330] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde: stopping tablet replica
I20260925 12:20:58.150776 20557 raft_consensus.cc:2229] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:20:58.150875 20557 raft_consensus.cc:2258] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:20:58.150980 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 20129
I20260925 12:20:58.151036 20556 raft_consensus.cc:2229] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:20:58.151616 20556 raft_consensus.cc:2258] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:20:58.152050 20557 ts_tablet_manager.cc:1905] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:20:58.153570 20556 ts_tablet_manager.cc:1905] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:20:58.154062 20557 ts_tablet_manager.cc:1918] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:20:58.154179 20557 log.cc:1196] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-2/wal/wals/f63265c917d642ebbd861c17ea08b69e
I20260925 12:20:58.154543 20557 ts_tablet_manager.cc:1939] T f63265c917d642ebbd861c17ea08b69e P b6504e1a7e114f4e9bcc6238b2751dde: Deleting consensus metadata
I20260925 12:20:58.154964 20053 catalog_manager.cc:4841] TS b6504e1a7e114f4e9bcc6238b2751dde (127.0.98.3:39631): tablet f63265c917d642ebbd861c17ea08b69e (table test-table [id=b44c57457d4e4fc5b16e2bc993dde548]) successfully deleted
I20260925 12:20:58.155148 20556 ts_tablet_manager.cc:1918] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:20:58.155217 20556 log.cc:1196] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe: Deleting WAL directory at /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338625091267-392-0/minicluster-data/ts-1/wal/wals/f63265c917d642ebbd861c17ea08b69e
I20260925 12:20:58.155472 20556 ts_tablet_manager.cc:1939] T f63265c917d642ebbd861c17ea08b69e P 14cdf173d0bd4f109a840f4b095fdabe: Deleting consensus metadata
I20260925 12:20:58.155930 20055 catalog_manager.cc:4841] TS 14cdf173d0bd4f109a840f4b095fdabe (127.0.98.2:33907): tablet f63265c917d642ebbd861c17ea08b69e (table test-table [id=b44c57457d4e4fc5b16e2bc993dde548]) successfully deleted
W20260925 12:20:58.157967 20056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33119 (error 108)
I20260925 12:20:58.158195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 20260
I20260925 12:20:58.158389 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:58.158277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33119 (local address 127.0.0.1:44394)
0925 12:20:58.158323 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 12:20:58.158358 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 12:20:58.158725 20056 catalog_manager.cc:4584] TS 011cc2d6eae842cb8b4faea2a687125f (127.0.98.1:33119): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f63265c917d642ebbd861c17ea08b69e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33119: connect: Connection refused (error 111)
I20260925 12:20:58.163224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 20391
I20260925 12:20:58.168704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 20038
2026-09-25T12:20:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (805 ms)
[----------] 6 tests from EncryptionPolicyTest (8382 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:20:58 dist-test-slave-clmx krb5kdc[20562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:20:58 dist-test-slave-clmx krb5kdc[20562](info): set up 2 sockets
Sep 25 12:20:58 dist-test-slave-clmx krb5kdc[20562](info): commencing operation
krb5kdc: starting...
W20260925 12:20:59.155460 392 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:20:59 dist-test-slave-clmx krb5kdc[20562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:20:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:20:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:20:59.264346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40025
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:20:59.342787 20575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:20:59.343024 20575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:20:59.343065 20575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:20:59.344208 20575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:20:59.344276 20575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:20:59.344301 20575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:20:59.344329 20575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:20:59.344360 20575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:20:59.345777 20575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40025
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:20:59.346035 20575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:20:59.346257 20575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:20:59.350809 20580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:59.350803 20583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:20:59.350804 20581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:20:59.351114 20575 server_base.cc:1034] running on GCE node
I20260925 12:20:59.351323 20575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:20:59.351573 20575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:20:59.352735 20575 hybrid_clock.cc:648] HybridClock initialized: now 1790338859352736 us; error 49 us; skew 500 ppm
Sep 25 12:20:59 dist-test-slave-clmx krb5kdc[20562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:20:59.355122 20575 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:20:59.355480 20575 webserver.cc:460] Webserver started at http://127.0.0.1:38757/ using document root <none> and password file <none>
I20260925 12:20:59.355734 20575 fs_manager.cc:359] Metadata directory not provided
I20260925 12:20:59.355777 20575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:20:59.355911 20575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:20:59.356786 20575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "1315fb769c4e45ae875acede7abd1089"
format_stamp: "Formatted at 2026-09-25 12:20:59 on dist-test-slave-clmx"
server_key: "c5d079282e772952aee4e3ed10d7b176"
server_key_iv: "dc556d20404b2a1f55426245d71741d9"
server_key_version: "encryptionkey@0"
I20260925 12:20:59.357105 20575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "1315fb769c4e45ae875acede7abd1089"
format_stamp: "Formatted at 2026-09-25 12:20:59 on dist-test-slave-clmx"
server_key: "c5d079282e772952aee4e3ed10d7b176"
server_key_iv: "dc556d20404b2a1f55426245d71741d9"
server_key_version: "encryptionkey@0"
I20260925 12:20:59.358314 20575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:20:59.359062 20590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:20:59.359228 20575 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:20:59.359308 20575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "1315fb769c4e45ae875acede7abd1089"
format_stamp: "Formatted at 2026-09-25 12:20:59 on dist-test-slave-clmx"
server_key: "c5d079282e772952aee4e3ed10d7b176"
server_key_iv: "dc556d20404b2a1f55426245d71741d9"
server_key_version: "encryptionkey@0"
I20260925 12:20:59.359368 20575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:20:59.375857 20575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:20:59.377842 20575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:20:59.377939 20575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:20:59.381563 20575 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40025
I20260925 12:20:59.381608 20642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40025 every 8 connection(s)
I20260925 12:20:59.381894 20575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:20:59.382648 20643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:20:59.384688 20643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089: Bootstrap starting.
I20260925 12:20:59.385211 20643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089: Neither blocks nor log segments found. Creating new log.
I20260925 12:20:59.385517 20643 log.cc:824] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089: Log is configured to *not* fsync() on all Append() calls
I20260925 12:20:59.386138 20643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089: No bootstrap required, opened a new log
I20260925 12:20:59.387591 20643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1315fb769c4e45ae875acede7abd1089" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40025 } }
I20260925 12:20:59.387782 20643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:20:59.387826 20643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1315fb769c4e45ae875acede7abd1089, State: Initialized, Role: FOLLOWER
I20260925 12:20:59.388003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 20575
I20260925 12:20:59.388000 20643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1315fb769c4e45ae875acede7abd1089" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40025 } }
I20260925 12:20:59.388101 20643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:20:59.388123 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:20:59.388150 20643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:20:59.388197 20643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:20:59.388310 392 external_mini_cluster.cc:1428] Setting key effa5302045d037884cec9c73afd9b5c
I20260925 12:20:59.388866 20643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1315fb769c4e45ae875acede7abd1089" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40025 } }
I20260925 12:20:59.388978 20643 leader_election.cc:302] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1315fb769c4e45ae875acede7abd1089; no voters:
I20260925 12:20:59.389114 20643 leader_election.cc:288] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:20:59.389166 20648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:20:59.389297 20648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089 [term 1 LEADER]: Becoming Leader. State: Replica: 1315fb769c4e45ae875acede7abd1089, State: Running, Role: LEADER
I20260925 12:20:59.389339 20643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:20:59.389386 20648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1315fb769c4e45ae875acede7abd1089" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40025 } }
I20260925 12:20:59.389889 20648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1315fb769c4e45ae875acede7abd1089. Latest consensus state: current_term: 1 leader_uuid: "1315fb769c4e45ae875acede7abd1089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1315fb769c4e45ae875acede7abd1089" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40025 } } }
I20260925 12:20:59.389963 20648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089 [sys.catalog]: This master's current role is: LEADER
I20260925 12:20:59.390101 20649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1315fb769c4e45ae875acede7abd1089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1315fb769c4e45ae875acede7abd1089" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40025 } } }
I20260925 12:20:59.390182 20649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089 [sys.catalog]: This master's current role is: LEADER
I20260925 12:20:59.390318 20657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:20:59.390898 20657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:20:59.392302 20657 catalog_manager.cc:1347] Generated new cluster ID: 2bed2859709249c888ed27596f638d2e
I20260925 12:20:59.392354 20657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:20:59 dist-test-slave-clmx krb5kdc[20562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 12:20:59.399600 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:59.388730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33241 (local address 127.0.0.1:40025)
0925 12:20:59.388881 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:59.388890 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:59.388920 (+ 30us) server_negotiation.cc:407] Connection header received
0925 12:20:59.389024 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:59.389031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:59.389085 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:59.389221 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:59.390129 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:59.391681 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:59.392671 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:59.393251 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:59.393356 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:59.395967 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:59.395970 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:59.395971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:59.395984 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:59.397910 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:59.398327 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:59.398330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:59.398331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:59.398454 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:59.398677 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:59.398678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:59.398678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:59.398968 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:59.399128 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:59.399208 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 12:20:59.399417 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260925 12:20:59.400853 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:20:59.408741 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:20:59.402141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41045 (local address 127.0.0.1:40025)
0925 12:20:59.402174 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 12:20:59.402175 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:20:59.402375 (+ 200us) server_negotiation.cc:407] Connection header received
0925 12:20:59.402397 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:20:59.402398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:20:59.402407 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:20:59.402470 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:20:59.402989 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:59.403740 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:59.404704 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:20:59.405028 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:20:59.405096 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:20:59.406271 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:20:59.406272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:20:59.406272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:20:59.406278 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:20:59.407697 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:59.408050 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:59.408051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:59.408051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:59.408102 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:20:59.408300 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:20:59.408301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:20:59.408301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:20:59.408417 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:20:59.408503 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 12:20:59.408616 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 12:20:59.408668 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:20:59.411054 20657 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:20:59.411514 20657 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:20:59.421312 20657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1315fb769c4e45ae875acede7abd1089: Generated new TSK 0
I20260925 12:20:59.421492 20657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:20:59.424544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 20575
W20260925 12:20:59.428524 20560 connection.cc:535] client connection to 127.0.0.1:40025 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40025): Connection reset by peer (error 104)
2026-09-25T12:20:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1254 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:20:59 dist-test-slave-clmx krb5kdc[20674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:20:59 dist-test-slave-clmx krb5kdc[20674](info): set up 2 sockets
Sep 25 12:20:59 dist-test-slave-clmx krb5kdc[20674](info): commencing operation
krb5kdc: starting...
W20260925 12:21:00.419445 392 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:21:00 dist-test-slave-clmx krb5kdc[20674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:21:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:21:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:21:00.526413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34875
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:21:00.602026 20687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:00.602187 20687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:00.602206 20687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:00.603219 20687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:00.603250 20687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:00.603263 20687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:00.603276 20687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:00.603286 20687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:00.604719 20687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34875
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:00.604890 20687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:00.605057 20687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:00.609403 20693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:00.609412 20695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:00.609411 20692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:00.609563 20687 server_base.cc:1034] running on GCE node
I20260925 12:21:00.609787 20687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:00.610065 20687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:00.611223 20687 hybrid_clock.cc:648] HybridClock initialized: now 1790338860611204 us; error 49 us; skew 500 ppm
Sep 25 12:21:00 dist-test-slave-clmx krb5kdc[20674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:00.619864 20687 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:21:00.620287 20687 webserver.cc:460] Webserver started at http://127.0.0.1:43293/ using document root <none> and password file <none>
I20260925 12:21:00.620537 20687 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:00.620602 20687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:00.620746 20687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:00.621729 20687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "81e7edb8d2aa4996a994b7c32dd4b939"
format_stamp: "Formatted at 2026-09-25 12:21:00 on dist-test-slave-clmx"
server_key: "4fc554ce4b449cb2406d9b8c098bcc56"
server_key_iv: "8a4b6b6b58ea56dbce20fef002a27822"
server_key_version: "encryptionkey@0"
I20260925 12:21:00.622054 20687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "81e7edb8d2aa4996a994b7c32dd4b939"
format_stamp: "Formatted at 2026-09-25 12:21:00 on dist-test-slave-clmx"
server_key: "4fc554ce4b449cb2406d9b8c098bcc56"
server_key_iv: "8a4b6b6b58ea56dbce20fef002a27822"
server_key_version: "encryptionkey@0"
I20260925 12:21:00.623291 20687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:21:00.624017 20702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:00.624177 20687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:00.624269 20687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "81e7edb8d2aa4996a994b7c32dd4b939"
format_stamp: "Formatted at 2026-09-25 12:21:00 on dist-test-slave-clmx"
server_key: "4fc554ce4b449cb2406d9b8c098bcc56"
server_key_iv: "8a4b6b6b58ea56dbce20fef002a27822"
server_key_version: "encryptionkey@0"
I20260925 12:21:00.624358 20687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:00.655979 20687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:00.657618 20687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:00.657761 20687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:00.661739 20687 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34875
I20260925 12:21:00.661770 20754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34875 every 8 connection(s)
I20260925 12:21:00.662081 20687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:21:00.662902 20755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:00.664921 20755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939: Bootstrap starting.
I20260925 12:21:00.665405 20755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:00.665627 20755 log.cc:824] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:00.666208 20755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939: No bootstrap required, opened a new log
I20260925 12:21:00.667606 20755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e7edb8d2aa4996a994b7c32dd4b939" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34875 } }
I20260925 12:21:00.667755 20755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:21:00.667776 20755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e7edb8d2aa4996a994b7c32dd4b939, State: Initialized, Role: FOLLOWER
I20260925 12:21:00.667896 20755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e7edb8d2aa4996a994b7c32dd4b939" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34875 } }
I20260925 12:21:00.667974 20755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:21:00.667997 20755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:21:00.668023 20755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:00.668445 20755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e7edb8d2aa4996a994b7c32dd4b939" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34875 } }
I20260925 12:21:00.668519 20755 leader_election.cc:302] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81e7edb8d2aa4996a994b7c32dd4b939; no voters:
I20260925 12:21:00.668619 20755 leader_election.cc:288] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:21:00.668686 20758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:21:00.668833 20758 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939 [term 1 LEADER]: Becoming Leader. State: Replica: 81e7edb8d2aa4996a994b7c32dd4b939, State: Running, Role: LEADER
I20260925 12:21:00.668865 20755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:00.668916 20758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e7edb8d2aa4996a994b7c32dd4b939" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34875 } }
I20260925 12:21:00.669260 20759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81e7edb8d2aa4996a994b7c32dd4b939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e7edb8d2aa4996a994b7c32dd4b939" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34875 } } }
I20260925 12:21:00.669284 20760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e7edb8d2aa4996a994b7c32dd4b939. Latest consensus state: current_term: 1 leader_uuid: "81e7edb8d2aa4996a994b7c32dd4b939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e7edb8d2aa4996a994b7c32dd4b939" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34875 } } }
I20260925 12:21:00.669445 20760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939 [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:00.669394 20759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939 [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:00.669734 20762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:21:00.670234 20762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:21:00.670792 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 20687
I20260925 12:21:00.670924 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:21:00.671105 392 external_mini_cluster.cc:1428] Setting key 65ef7ee4616eb6986a47b1a623a1e67c
I20260925 12:21:00.671924 20762 catalog_manager.cc:1347] Generated new cluster ID: 204da8d660724acda44cb84052b1c17d
I20260925 12:21:00.671975 20762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:21:00.676705 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:00.671512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43337 (local address 127.0.0.1:34875)
0925 12:21:00.671705 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:00.671716 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:00.671769 (+ 53us) server_negotiation.cc:407] Connection header received
0925 12:21:00.671916 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:00.671922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:00.671976 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:00.672103 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:00.672327 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:00.673700 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:00.674756 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:00.675292 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:00.675377 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:00.676046 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:00.676050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:00.676050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:00.676257 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:00.676290 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:00.676303 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:00.676375 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:00.676504 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 12:21:00.676527 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":143,"thread_start_us":68,"threads_started":1}
I20260925 12:21:00.677537 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:21:00.680264 20762 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:21:00.680738 20762 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:21:00.683262 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:00.678937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58157 (local address 127.0.0.1:34875)
0925 12:21:00.678971 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:00.678972 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:00.679925 (+ 953us) server_negotiation.cc:407] Connection header received
0925 12:21:00.680019 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:00.680020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:00.680031 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:00.680091 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:00.680240 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:00.680987 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:00.682007 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:00.682352 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:00.682419 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:00.683027 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:00.683028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:00.683028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:00.683033 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:00.683045 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:00.683050 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:00.683109 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:00.683175 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 12:21:00.683186 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:21:00.690768 20762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81e7edb8d2aa4996a994b7c32dd4b939: Generated new TSK 0
I20260925 12:21:00.690927 20762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:21:00.695964 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 20687
W20260925 12:21:00.700057 20672 connection.cc:535] client connection to 127.0.0.1:34875 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34875): Connection reset by peer (error 104)
2026-09-25T12:21:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1272 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:21:00 dist-test-slave-clmx krb5kdc[20786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:21:00 dist-test-slave-clmx krb5kdc[20786](info): set up 2 sockets
Sep 25 12:21:00 dist-test-slave-clmx krb5kdc[20786](info): commencing operation
krb5kdc: starting...
W20260925 12:21:01.691244 392 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:21:01 dist-test-slave-clmx krb5kdc[20786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:21:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:21:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:21:01.798112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45745
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:21:01.873081 20799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:01.873296 20799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:01.873329 20799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:01.874464 20799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:01.874526 20799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:01.874549 20799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:01.874576 20799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:01.874607 20799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:01.876001 20799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45745
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:01.876246 20799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:01.876487 20799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:01.880826 20804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:01.880853 20805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:01.880853 20807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:01.880882 20799 server_base.cc:1034] running on GCE node
I20260925 12:21:01.881192 20799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:01.881448 20799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:01.882578 20799 hybrid_clock.cc:648] HybridClock initialized: now 1790338861882580 us; error 39 us; skew 500 ppm
Sep 25 12:21:01 dist-test-slave-clmx krb5kdc[20786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:01.884857 20799 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:21:01.885252 20799 webserver.cc:460] Webserver started at http://127.0.0.1:33323/ using document root <none> and password file <none>
I20260925 12:21:01.885468 20799 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:01.885531 20799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:01.885667 20799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:01.886633 20799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "5a7ce07ce3c445c6823f6a1dc46174cc"
format_stamp: "Formatted at 2026-09-25 12:21:01 on dist-test-slave-clmx"
server_key: "e81bb3a94086f252cc5315d6d5c4ef6a"
server_key_iv: "a75403eda7c6ad67e9da553b960f1e16"
server_key_version: "encryptionkey@0"
I20260925 12:21:01.886973 20799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "5a7ce07ce3c445c6823f6a1dc46174cc"
format_stamp: "Formatted at 2026-09-25 12:21:01 on dist-test-slave-clmx"
server_key: "e81bb3a94086f252cc5315d6d5c4ef6a"
server_key_iv: "a75403eda7c6ad67e9da553b960f1e16"
server_key_version: "encryptionkey@0"
I20260925 12:21:01.888239 20799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 12:21:01.888865 20814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:01.889026 20799 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:21:01.889118 20799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "5a7ce07ce3c445c6823f6a1dc46174cc"
format_stamp: "Formatted at 2026-09-25 12:21:01 on dist-test-slave-clmx"
server_key: "e81bb3a94086f252cc5315d6d5c4ef6a"
server_key_iv: "a75403eda7c6ad67e9da553b960f1e16"
server_key_version: "encryptionkey@0"
I20260925 12:21:01.889189 20799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:01.892287 20799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:01.893883 20799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:01.894033 20799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:01.897682 20799 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45745
I20260925 12:21:01.897709 20866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45745 every 8 connection(s)
I20260925 12:21:01.897977 20799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:21:01.898808 20867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:01.900939 20867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc: Bootstrap starting.
I20260925 12:21:01.901470 20867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:01.901751 20867 log.cc:824] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:01.901974 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 20799
I20260925 12:21:01.902091 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:21:01.902272 392 external_mini_cluster.cc:1428] Setting key c23199836aacd878e6793ffcffeec540
I20260925 12:21:01.902437 20867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc: No bootstrap required, opened a new log
I20260925 12:21:01.904145 20867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a7ce07ce3c445c6823f6a1dc46174cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45745 } }
I20260925 12:21:01.904297 20867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:21:01.904337 20867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a7ce07ce3c445c6823f6a1dc46174cc, State: Initialized, Role: FOLLOWER
I20260925 12:21:01.904461 20867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a7ce07ce3c445c6823f6a1dc46174cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45745 } }
I20260925 12:21:01.904580 20867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:21:01.904625 20867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:21:01.904680 20867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:01.905416 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:01.902861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51699 (local address 127.0.0.1:45745)
0925 12:21:01.903029 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:01.903036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:01.903067 (+ 31us) server_negotiation.cc:407] Connection header received
0925 12:21:01.904129 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:01.904134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:01.904158 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:01.904264 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:01.904957 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:01.904960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:01.904960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:01.904963 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:01.904997 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:01.905007 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:01.905060 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:01.905154 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 12:21:01.905182 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260925 12:21:01.905761 20867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a7ce07ce3c445c6823f6a1dc46174cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45745 } }
I20260925 12:21:01.905881 20867 leader_election.cc:302] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a7ce07ce3c445c6823f6a1dc46174cc; no voters:
I20260925 12:21:01.906033 20867 leader_election.cc:288] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:21:01.906060 20872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:21:01.906160 20872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc [term 1 LEADER]: Becoming Leader. State: Replica: 5a7ce07ce3c445c6823f6a1dc46174cc, State: Running, Role: LEADER
I20260925 12:21:01.906263 20867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:01.906258 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:21:01.906232 20872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a7ce07ce3c445c6823f6a1dc46174cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45745 } }
I20260925 12:21:01.906555 20873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a7ce07ce3c445c6823f6a1dc46174cc. Latest consensus state: current_term: 1 leader_uuid: "5a7ce07ce3c445c6823f6a1dc46174cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a7ce07ce3c445c6823f6a1dc46174cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45745 } } }
I20260925 12:21:01.906756 20873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:01.906695 20874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a7ce07ce3c445c6823f6a1dc46174cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a7ce07ce3c445c6823f6a1dc46174cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45745 } } }
I20260925 12:21:01.906993 20874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:01.907143 20881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:21:01.907676 20881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:21:01.908941 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:01.907702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36617 (local address 127.0.0.1:45745)
0925 12:21:01.907735 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:01.907737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:01.908022 (+ 285us) server_negotiation.cc:407] Connection header received
0925 12:21:01.908109 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:01.908110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:01.908114 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:01.908173 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:01.908712 (+ 539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:01.908713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:01.908713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:01.908717 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:01.908729 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:01.908733 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:01.908810 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:01.908859 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 12:21:01.908872 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:21:01.909607 20881 catalog_manager.cc:1347] Generated new cluster ID: 728a0d01842140deb663db332994f990
I20260925 12:21:01.909657 20881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:21:01.916122 20881 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:21:01.916260 20881 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:21:01.925725 20881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a7ce07ce3c445c6823f6a1dc46174cc: Generated new TSK 0
I20260925 12:21:01.925889 20881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:21:01.930061 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 20799
2026-09-25T12:21:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1232 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:21:01 dist-test-slave-clmx krb5kdc[20898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:21:01 dist-test-slave-clmx krb5kdc[20898](info): set up 2 sockets
Sep 25 12:21:01 dist-test-slave-clmx krb5kdc[20898](info): commencing operation
krb5kdc: starting...
W20260925 12:21:02.927379 392 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:21:02 dist-test-slave-clmx krb5kdc[20898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:21:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:21:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:21:03.030432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37191
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:21:03.103267 20911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:03.103472 20911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:03.103507 20911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:03.104632 20911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:03.104697 20911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:03.104722 20911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:03.104749 20911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:03.104779 20911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:03.106158 20911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:03.106402 20911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:03.106658 20911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:03.111398 20916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:03.111438 20917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:03.111399 20919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:03.111575 20911 server_base.cc:1034] running on GCE node
I20260925 12:21:03.111857 20911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:03.112124 20911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:03.113257 20911 hybrid_clock.cc:648] HybridClock initialized: now 1790338863113248 us; error 30 us; skew 500 ppm
Sep 25 12:21:03 dist-test-slave-clmx krb5kdc[20898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:03.115518 20911 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:21:03.116012 20911 webserver.cc:460] Webserver started at http://127.0.0.1:45967/ using document root <none> and password file <none>
I20260925 12:21:03.116240 20911 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:03.116295 20911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:03.116452 20911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:03.117787 20911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "da69910c3c614827825ac12109846aa7"
format_stamp: "Formatted at 2026-09-25 12:21:03 on dist-test-slave-clmx"
server_key: "caafd5c00421adcae2ffd8cb37cc22bd"
server_key_iv: "d9528f8b165285462587bf1b710aa599"
server_key_version: "encryptionkey@0"
I20260925 12:21:03.118166 20911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "da69910c3c614827825ac12109846aa7"
format_stamp: "Formatted at 2026-09-25 12:21:03 on dist-test-slave-clmx"
server_key: "caafd5c00421adcae2ffd8cb37cc22bd"
server_key_iv: "d9528f8b165285462587bf1b710aa599"
server_key_version: "encryptionkey@0"
I20260925 12:21:03.119680 20911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:21:03.120463 20926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:03.120612 20911 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:03.120677 20911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "da69910c3c614827825ac12109846aa7"
format_stamp: "Formatted at 2026-09-25 12:21:03 on dist-test-slave-clmx"
server_key: "caafd5c00421adcae2ffd8cb37cc22bd"
server_key_iv: "d9528f8b165285462587bf1b710aa599"
server_key_version: "encryptionkey@0"
I20260925 12:21:03.120736 20911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:03.133857 20911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:03.135879 20911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:03.136021 20911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:03.139505 20911 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37191
I20260925 12:21:03.139568 20978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37191 every 8 connection(s)
I20260925 12:21:03.139863 20911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:21:03.140794 20979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:03.143934 20979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7: Bootstrap starting.
I20260925 12:21:03.144277 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 20911
I20260925 12:21:03.144399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:21:03.144595 392 external_mini_cluster.cc:1428] Setting key e085ffea2e0b87e0c8d5f2e11de60897
I20260925 12:21:03.144627 20979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:03.144994 20979 log.cc:824] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:03.145716 20979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7: No bootstrap required, opened a new log
I20260925 12:21:03.147904 20979 raft_consensus.cc:348] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da69910c3c614827825ac12109846aa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37191 } }
I20260925 12:21:03.148052 20979 raft_consensus.cc:374] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:21:03.148080 20979 raft_consensus.cc:729] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da69910c3c614827825ac12109846aa7, State: Initialized, Role: FOLLOWER
I20260925 12:21:03.148186 20979 consensus_queue.cc:260] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da69910c3c614827825ac12109846aa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37191 } }
I20260925 12:21:03.148248 20979 raft_consensus.cc:388] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:21:03.148275 20979 raft_consensus.cc:482] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:21:03.148317 20979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:03.148885 20979 raft_consensus.cc:504] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da69910c3c614827825ac12109846aa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37191 } }
I20260925 12:21:03.148959 20979 leader_election.cc:302] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da69910c3c614827825ac12109846aa7; no voters:
I20260925 12:21:03.149066 20979 leader_election.cc:288] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:21:03.149117 20984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:21:03.149251 20984 raft_consensus.cc:686] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7 [term 1 LEADER]: Becoming Leader. State: Replica: da69910c3c614827825ac12109846aa7, State: Running, Role: LEADER
I20260925 12:21:03.149295 20979 sys_catalog.cc:564] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:03.149322 20984 consensus_queue.cc:237] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da69910c3c614827825ac12109846aa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37191 } }
I20260925 12:21:03.149688 20986 sys_catalog.cc:455] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da69910c3c614827825ac12109846aa7. Latest consensus state: current_term: 1 leader_uuid: "da69910c3c614827825ac12109846aa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da69910c3c614827825ac12109846aa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37191 } } }
I20260925 12:21:03.149827 20986 sys_catalog.cc:458] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7 [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:03.150067 20993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:21:03.150050 20985 sys_catalog.cc:455] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da69910c3c614827825ac12109846aa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da69910c3c614827825ac12109846aa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37191 } } }
I20260925 12:21:03.150197 20985 sys_catalog.cc:458] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7 [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:03.150544 20993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 12:21:03 dist-test-slave-clmx krb5kdc[20898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 12:21:03.153204 21001 mvcc.cc:205] Tried to move back new op lower bound from 7333227983470227456 to 7333227983460032512. Current Snapshot: MvccSnapshot[applied={T|T < 7333227983470227456}]
I20260925 12:21:03.157575 20982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:03.145048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48699 (local address 127.0.0.1:37191)
0925 12:21:03.145213 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:03.145221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:03.145254 (+ 33us) server_negotiation.cc:407] Connection header received
0925 12:21:03.145876 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:03.145884 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:03.145955 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:03.146091 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:03.146892 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:03.148284 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:03.149407 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:03.150062 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:03.150161 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:03.153287 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:03.153291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:03.153292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:03.153303 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:03.155453 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:03.155891 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:03.155895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:03.155895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:03.156008 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:03.156228 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:03.156229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:03.156229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:03.156554 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:03.157115 (+ 561us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:03.157190 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 12:21:03.157351 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
I20260925 12:21:03.158579 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:21:03.164044 20993 catalog_manager.cc:1347] Generated new cluster ID: e3d3409c66f64cd3838f843cf7db44d3
I20260925 12:21:03.164114 20993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:21:03.169930 20982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:03.160137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47205 (local address 127.0.0.1:37191)
0925 12:21:03.160172 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:03.160174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:03.163830 (+ 3656us) server_negotiation.cc:407] Connection header received
0925 12:21:03.163873 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:03.163874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:03.163883 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:03.163938 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:03.164612 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:03.165320 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:03.166196 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:03.166468 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:03.166498 (+ 30us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:03.167655 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:03.167656 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:03.167657 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:03.167662 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:03.168961 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:03.169260 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:03.169261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:03.169261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:03.169301 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:03.169481 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:03.169482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:03.169482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:03.169577 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:03.169631 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:03.169794 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 12:21:03.169854 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:21:03.180160 20993 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:21:03.180656 20993 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:21:03.183759 20993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da69910c3c614827825ac12109846aa7: Generated new TSK 0
I20260925 12:21:03.183957 20993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:21:03.186266 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 20911
2026-09-25T12:21:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1257 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:21:03 dist-test-slave-clmx krb5kdc[21010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:21:03 dist-test-slave-clmx krb5kdc[21010](info): set up 2 sockets
Sep 25 12:21:03 dist-test-slave-clmx krb5kdc[21010](info): commencing operation
krb5kdc: starting...
W20260925 12:21:04.179401 392 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:21:04 dist-test-slave-clmx krb5kdc[21010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:21:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:21:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:21:04.284456 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43215
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:21:04.359272 21023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:04.359470 21023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:04.359501 21023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:04.360620 21023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:04.360684 21023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:04.360726 21023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:04.360754 21023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:04.360785 21023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:04.362179 21023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:04.362412 21023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:04.362607 21023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:21:04.367115 21023 server_base.cc:1034] running on GCE node
W20260925 12:21:04.367079 21028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:04.367102 21029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:04.367075 21031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:04.367436 21023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:04.367750 21023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:04.368904 21023 hybrid_clock.cc:648] HybridClock initialized: now 1790338864368898 us; error 40 us; skew 500 ppm
Sep 25 12:21:04 dist-test-slave-clmx krb5kdc[21010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:04.371219 21023 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:21:04.371699 21023 webserver.cc:460] Webserver started at http://127.0.0.1:36793/ using document root <none> and password file <none>
I20260925 12:21:04.371932 21023 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:04.371989 21023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:04.372123 21023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:04.373041 21023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "6102f659f2cc498d9474ef43297262e1"
format_stamp: "Formatted at 2026-09-25 12:21:04 on dist-test-slave-clmx"
server_key: "cc231e1b066dbda983c7c4635ae26b97"
server_key_iv: "3f3bc71691adce4d095370b1a49606bf"
server_key_version: "encryptionkey@0"
I20260925 12:21:04.373371 21023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "6102f659f2cc498d9474ef43297262e1"
format_stamp: "Formatted at 2026-09-25 12:21:04 on dist-test-slave-clmx"
server_key: "cc231e1b066dbda983c7c4635ae26b97"
server_key_iv: "3f3bc71691adce4d095370b1a49606bf"
server_key_version: "encryptionkey@0"
I20260925 12:21:04.374569 21023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:21:04.375293 21038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:04.375461 21023 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:04.375551 21023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "6102f659f2cc498d9474ef43297262e1"
format_stamp: "Formatted at 2026-09-25 12:21:04 on dist-test-slave-clmx"
server_key: "cc231e1b066dbda983c7c4635ae26b97"
server_key_iv: "3f3bc71691adce4d095370b1a49606bf"
server_key_version: "encryptionkey@0"
I20260925 12:21:04.375655 21023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:04.413156 21023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:04.415313 21023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:04.415498 21023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:04.419006 21023 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43215
I20260925 12:21:04.419044 21090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43215 every 8 connection(s)
I20260925 12:21:04.419365 21023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:21:04.420163 21091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:04.422284 21091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1: Bootstrap starting.
I20260925 12:21:04.422825 21091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:04.423092 21091 log.cc:824] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:04.423774 21091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1: No bootstrap required, opened a new log
I20260925 12:21:04.425305 21091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6102f659f2cc498d9474ef43297262e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43215 } }
I20260925 12:21:04.425446 21091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:21:04.425468 21091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6102f659f2cc498d9474ef43297262e1, State: Initialized, Role: FOLLOWER
I20260925 12:21:04.425645 21091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6102f659f2cc498d9474ef43297262e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43215 } }
I20260925 12:21:04.425704 21091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:21:04.425755 21091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:21:04.425802 21091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:04.426286 21091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6102f659f2cc498d9474ef43297262e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43215 } }
I20260925 12:21:04.426395 21091 leader_election.cc:302] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6102f659f2cc498d9474ef43297262e1; no voters:
I20260925 12:21:04.426568 21091 leader_election.cc:288] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:21:04.426617 21094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:21:04.426751 21094 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1 [term 1 LEADER]: Becoming Leader. State: Replica: 6102f659f2cc498d9474ef43297262e1, State: Running, Role: LEADER
I20260925 12:21:04.426795 21091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:04.426836 21094 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6102f659f2cc498d9474ef43297262e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43215 } }
I20260925 12:21:04.427193 21096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6102f659f2cc498d9474ef43297262e1. Latest consensus state: current_term: 1 leader_uuid: "6102f659f2cc498d9474ef43297262e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6102f659f2cc498d9474ef43297262e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43215 } } }
I20260925 12:21:04.427274 21096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1 [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:04.427407 21095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6102f659f2cc498d9474ef43297262e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6102f659f2cc498d9474ef43297262e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43215 } } }
I20260925 12:21:04.427495 21095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1 [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:04.427678 21102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:21:04.428208 21102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:21:04.428562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 21023
I20260925 12:21:04.428681 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:21:04.428869 392 external_mini_cluster.cc:1428] Setting key e60934312c479783a9edee4970c841bd
I20260925 12:21:04.432087 21102 catalog_manager.cc:1347] Generated new cluster ID: 2fda4c51b67b4b409041926448884e79
I20260925 12:21:04.432137 21102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:21:04.434389 21113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:04.429253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48803 (local address 127.0.0.1:43215)
0925 12:21:04.429631 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:04.429640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:04.429670 (+ 30us) server_negotiation.cc:407] Connection header received
0925 12:21:04.429780 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:04.429787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:04.429835 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:04.430089 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:04.430173 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:04.431573 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:04.432567 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:04.432969 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:04.433092 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:04.433728 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:04.433731 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:04.433731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:04.433919 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:04.433947 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:04.433957 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:04.434028 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:04.434099 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 12:21:04.434237 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":64,"threads_started":1}
I20260925 12:21:04.435309 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:21:04.442313 21113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:04.436732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60957 (local address 127.0.0.1:43215)
0925 12:21:04.436768 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:04.436770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:04.439771 (+ 3001us) server_negotiation.cc:407] Connection header received
0925 12:21:04.439836 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:04.439837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:04.439846 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:04.439900 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:04.440043 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:04.440474 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:04.441192 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:04.441403 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:04.441479 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:04.442094 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:04.442095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:04.442095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:04.442100 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:04.442112 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:04.442116 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:04.442159 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:04.442209 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 12:21:04.442228 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:21:04.444923 21102 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:21:04.445600 21102 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:21:04.452045 21102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6102f659f2cc498d9474ef43297262e1: Generated new TSK 0
I20260925 12:21:04.452190 21102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:21:04.458209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 21023
2026-09-25T12:21:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1271 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:21:04 dist-test-slave-clmx krb5kdc[21122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:21:04 dist-test-slave-clmx krb5kdc[21122](info): set up 2 sockets
Sep 25 12:21:04 dist-test-slave-clmx krb5kdc[21122](info): commencing operation
krb5kdc: starting...
W20260925 12:21:05.455545 392 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:21:05 dist-test-slave-clmx krb5kdc[21122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:21:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:21:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:21:05.569839 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:21:05.647063 21135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:05.647297 21135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:05.647332 21135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:05.648463 21135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:05.648541 21135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:05.648566 21135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:05.648587 21135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:05.648617 21135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:05.650054 21135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:05.650327 21135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:05.650535 21135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:05.655095 21143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:05.655103 21140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:05.655095 21141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:05.655481 21135 server_base.cc:1034] running on GCE node
I20260925 12:21:05.655709 21135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:05.655922 21135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:05.657048 21135 hybrid_clock.cc:648] HybridClock initialized: now 1790338865657038 us; error 30 us; skew 500 ppm
Sep 25 12:21:05 dist-test-slave-clmx krb5kdc[21122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:05.663587 21135 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:21:05.664008 21135 webserver.cc:460] Webserver started at http://127.0.0.1:34597/ using document root <none> and password file <none>
I20260925 12:21:05.664196 21135 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:05.664235 21135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:05.664316 21135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:05.665266 21135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "899f6694595042519094bc9456c3356c"
format_stamp: "Formatted at 2026-09-25 12:21:05 on dist-test-slave-clmx"
server_key: "80d0bd8c222f92fa2215102fe0da5af6"
server_key_iv: "e566d83424d40b1d9c832f39e452a0f5"
server_key_version: "encryptionkey@0"
I20260925 12:21:05.665584 21135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "899f6694595042519094bc9456c3356c"
format_stamp: "Formatted at 2026-09-25 12:21:05 on dist-test-slave-clmx"
server_key: "80d0bd8c222f92fa2215102fe0da5af6"
server_key_iv: "e566d83424d40b1d9c832f39e452a0f5"
server_key_version: "encryptionkey@0"
I20260925 12:21:05.666759 21135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:21:05.667363 21150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:05.667526 21135 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:05.667631 21135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "899f6694595042519094bc9456c3356c"
format_stamp: "Formatted at 2026-09-25 12:21:05 on dist-test-slave-clmx"
server_key: "80d0bd8c222f92fa2215102fe0da5af6"
server_key_iv: "e566d83424d40b1d9c832f39e452a0f5"
server_key_version: "encryptionkey@0"
I20260925 12:21:05.667712 21135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:05.670483 21135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:05.672297 21135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:05.672443 21135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:05.675874 21135 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42683
I20260925 12:21:05.675951 21202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42683 every 8 connection(s)
I20260925 12:21:05.676222 21135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:21:05.677138 21203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:05.679200 21203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c: Bootstrap starting.
I20260925 12:21:05.679814 21203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:05.680094 21203 log.cc:824] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:05.680716 21203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c: No bootstrap required, opened a new log
I20260925 12:21:05.682322 21203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899f6694595042519094bc9456c3356c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260925 12:21:05.682448 21203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:21:05.682497 21203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 899f6694595042519094bc9456c3356c, State: Initialized, Role: FOLLOWER
I20260925 12:21:05.682677 21203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899f6694595042519094bc9456c3356c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260925 12:21:05.682758 21203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:21:05.682814 21203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:21:05.682868 21203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:05.683372 21203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899f6694595042519094bc9456c3356c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260925 12:21:05.683482 21203 leader_election.cc:302] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 899f6694595042519094bc9456c3356c; no voters:
I20260925 12:21:05.683673 21203 leader_election.cc:288] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:21:05.683728 21206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:21:05.684062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 21135
I20260925 12:21:05.684198 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:21:05.684340 392 external_mini_cluster.cc:1428] Setting key aafa97a60805b8d0083f3a05caf070dc
I20260925 12:21:05.683869 21206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c [term 1 LEADER]: Becoming Leader. State: Replica: 899f6694595042519094bc9456c3356c, State: Running, Role: LEADER
I20260925 12:21:05.683929 21203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:05.684551 21206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899f6694595042519094bc9456c3356c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260925 12:21:05.685408 21207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "899f6694595042519094bc9456c3356c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899f6694595042519094bc9456c3356c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } } }
I20260925 12:21:05.685410 21208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 899f6694595042519094bc9456c3356c. Latest consensus state: current_term: 1 leader_uuid: "899f6694595042519094bc9456c3356c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899f6694595042519094bc9456c3356c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } } }
I20260925 12:21:05.685576 21207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:05.685592 21208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:05.686151 21220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:21:05.686642 21220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:21:05.686817 21215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:05.685045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53791 (local address 127.0.0.1:42683)
0925 12:21:05.685190 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:05.685199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:05.685229 (+ 30us) server_negotiation.cc:407] Connection header received
0925 12:21:05.685358 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:05.685366 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:05.685399 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:05.685793 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:05.686257 (+ 464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:05.686261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:05.686261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:05.686266 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:05.686303 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:05.686316 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:05.686378 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:05.686549 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 12:21:05.686587 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":84,"threads_started":1}
I20260925 12:21:05.687760 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:21:05.688174 21220 catalog_manager.cc:1347] Generated new cluster ID: 6fde9045fd6846dca048917682eb7e31
I20260925 12:21:05.688211 21220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:21:05.696877 21215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:05.691919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48703 (local address 127.0.0.1:42683)
0925 12:21:05.691979 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:05.691981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:05.695813 (+ 3832us) server_negotiation.cc:407] Connection header received
0925 12:21:05.695894 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:05.695895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:05.695902 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:05.695997 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:05.696642 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:05.696643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:05.696644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:05.696648 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:05.696662 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:05.696666 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:05.696727 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:05.696789 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 12:21:05.696803 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 12:21:05.697850 21220 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:21:05.697981 21220 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:21:05.707844 21220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 899f6694595042519094bc9456c3356c: Generated new TSK 0
I20260925 12:21:05.708105 21220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:21:05.711104 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 21135
2026-09-25T12:21:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1253 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:21:05 dist-test-slave-clmx krb5kdc[21234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:21:05 dist-test-slave-clmx krb5kdc[21234](info): set up 2 sockets
Sep 25 12:21:05 dist-test-slave-clmx krb5kdc[21234](info): commencing operation
krb5kdc: starting...
W20260925 12:21:06.695437 392 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:21:06 dist-test-slave-clmx krb5kdc[21234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:21:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:21:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:21:06.800657 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37329
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:21:06.875309 21247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:06.875520 21247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:06.875557 21247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:06.876673 21247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:06.876755 21247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:06.876778 21247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:06.876803 21247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:06.876834 21247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:06.878180 21247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:06.878427 21247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:06.878650 21247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:06.883107 21255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:06.883138 21253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:06.883268 21247 server_base.cc:1034] running on GCE node
W20260925 12:21:06.883111 21252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:06.883541 21247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:06.883817 21247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:06.884989 21247 hybrid_clock.cc:648] HybridClock initialized: now 1790338866884975 us; error 43 us; skew 500 ppm
Sep 25 12:21:06 dist-test-slave-clmx krb5kdc[21234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:06.887259 21247 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:21:06.887704 21247 webserver.cc:460] Webserver started at http://127.0.0.1:36363/ using document root <none> and password file <none>
I20260925 12:21:06.887924 21247 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:06.887988 21247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:06.888139 21247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:06.889415 21247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "8a55da39f45a4b73b9ff1033c6406d0c"
format_stamp: "Formatted at 2026-09-25 12:21:06 on dist-test-slave-clmx"
server_key: "5d9f2ade0d45f0788caa9c101fb876c9"
server_key_iv: "70fb477114a986d26d739dc15248b55f"
server_key_version: "encryptionkey@0"
I20260925 12:21:06.889760 21247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "8a55da39f45a4b73b9ff1033c6406d0c"
format_stamp: "Formatted at 2026-09-25 12:21:06 on dist-test-slave-clmx"
server_key: "5d9f2ade0d45f0788caa9c101fb876c9"
server_key_iv: "70fb477114a986d26d739dc15248b55f"
server_key_version: "encryptionkey@0"
I20260925 12:21:06.891067 21247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:21:06.891767 21262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:06.891927 21247 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:21:06.892025 21247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "8a55da39f45a4b73b9ff1033c6406d0c"
format_stamp: "Formatted at 2026-09-25 12:21:06 on dist-test-slave-clmx"
server_key: "5d9f2ade0d45f0788caa9c101fb876c9"
server_key_iv: "70fb477114a986d26d739dc15248b55f"
server_key_version: "encryptionkey@0"
I20260925 12:21:06.892108 21247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:06.905853 21247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:06.907964 21247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:06.908108 21247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:06.911809 21247 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37329
I20260925 12:21:06.911844 21314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37329 every 8 connection(s)
I20260925 12:21:06.912165 21247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:21:06.912951 21315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:06.914386 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 21247
I20260925 12:21:06.914511 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:21:06.914687 392 external_mini_cluster.cc:1428] Setting key 77b500f4276fda52a680b63a35925ce3
I20260925 12:21:06.915177 21315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c: Bootstrap starting.
I20260925 12:21:06.915809 21315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:06.916115 21315 log.cc:824] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:06.916851 21315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c: No bootstrap required, opened a new log
I20260925 12:21:06.918887 21315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a55da39f45a4b73b9ff1033c6406d0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37329 } }
I20260925 12:21:06.919021 21315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:21:06.919044 21315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a55da39f45a4b73b9ff1033c6406d0c, State: Initialized, Role: FOLLOWER
I20260925 12:21:06.919176 21315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a55da39f45a4b73b9ff1033c6406d0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37329 } }
I20260925 12:21:06.919229 21315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:21:06.919274 21315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:21:06.919322 21315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:06.920020 21315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a55da39f45a4b73b9ff1033c6406d0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37329 } }
I20260925 12:21:06.920130 21315 leader_election.cc:302] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a55da39f45a4b73b9ff1033c6406d0c; no voters:
I20260925 12:21:06.920284 21315 leader_election.cc:288] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:21:06.920310 21320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:21:06.920481 21320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c [term 1 LEADER]: Becoming Leader. State: Replica: 8a55da39f45a4b73b9ff1033c6406d0c, State: Running, Role: LEADER
I20260925 12:21:06.920559 21315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:06.920615 21320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a55da39f45a4b73b9ff1033c6406d0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37329 } }
I20260925 12:21:06.920907 21320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a55da39f45a4b73b9ff1033c6406d0c. Latest consensus state: current_term: 1 leader_uuid: "8a55da39f45a4b73b9ff1033c6406d0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a55da39f45a4b73b9ff1033c6406d0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37329 } } }
I20260925 12:21:06.920914 21322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a55da39f45a4b73b9ff1033c6406d0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a55da39f45a4b73b9ff1033c6406d0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37329 } } }
I20260925 12:21:06.921048 21320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:06.921056 21322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:06.921327 21324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 12:21:06 dist-test-slave-clmx krb5kdc[21234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 12:21:06.921777 21324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:21:06.923547 21324 catalog_manager.cc:1347] Generated new cluster ID: df833719a9404eaab15b55b69e92f982
I20260925 12:21:06.923595 21324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:21:06.925884 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:06.915179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:40635 (local address 127.0.0.1:37329)
0925 12:21:06.915359 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:06.915367 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:06.915389 (+ 22us) server_negotiation.cc:407] Connection header received
0925 12:21:06.915901 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:06.915909 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:06.915969 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:06.916084 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:06.916858 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:06.917739 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:06.918851 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:06.919386 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:06.919486 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:06.922209 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:06.922212 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:06.922212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:06.922221 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:06.924293 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:06.924723 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:06.924726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:06.924726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:06.924825 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:06.925050 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:06.925051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:06.925051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:06.925338 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:06.925482 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:06.925597 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 12:21:06.925690 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260925 12:21:06.926805 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:21:06.935169 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:06.928115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:40639 (local address 127.0.0.1:37329)
0925 12:21:06.928144 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:06.928146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:06.928396 (+ 250us) server_negotiation.cc:407] Connection header received
0925 12:21:06.928421 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:06.928422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:06.928431 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:06.928495 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:06.929005 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:06.929999 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:06.930921 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:06.931290 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:06.931364 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:06.932587 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:06.932588 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:06.932589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:06.932595 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:06.934039 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:06.934423 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:06.934424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:06.934424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:06.934479 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:06.934695 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:06.934696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:06.934697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:06.934811 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:06.934913 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:06.935028 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 12:21:06.935086 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 12:21:06.942371 21324 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:21:06.942884 21324 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:21:06.953747 21324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a55da39f45a4b73b9ff1033c6406d0c: Generated new TSK 0
I20260925 12:21:06.953895 21324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:21:06.957055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 21247
W20260925 12:21:06.960896 21232 connection.cc:535] client connection to 127.0.0.1:37329 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37329): Connection reset by peer (error 104)
2026-09-25T12:21:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1245 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:21:07 dist-test-slave-clmx krb5kdc[21346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:21:07 dist-test-slave-clmx krb5kdc[21346](info): set up 2 sockets
Sep 25 12:21:07 dist-test-slave-clmx krb5kdc[21346](info): commencing operation
krb5kdc: starting...
W20260925 12:21:07.935461 392 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:21:08 dist-test-slave-clmx krb5kdc[21346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:21:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:21:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:21:08.040258 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46671
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:21:08.114466 21359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:08.114647 21359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:08.114668 21359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:08.115768 21359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:08.115805 21359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:08.115819 21359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:08.115857 21359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:08.115883 21359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:08.117238 21359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:08.117461 21359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:08.117668 21359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:08.122063 21364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:08.122073 21365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:08.122068 21367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:08.122289 21359 server_base.cc:1034] running on GCE node
I20260925 12:21:08.122488 21359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:08.122740 21359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:08.123893 21359 hybrid_clock.cc:648] HybridClock initialized: now 1790338868123888 us; error 33 us; skew 500 ppm
Sep 25 12:21:08 dist-test-slave-clmx krb5kdc[21346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:08.126096 21359 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:21:08.126408 21359 webserver.cc:460] Webserver started at http://127.0.0.1:32775/ using document root <none> and password file <none>
I20260925 12:21:08.126581 21359 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:08.126616 21359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:08.126693 21359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:08.127555 21359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "06ecd32eec5c411f83e3c0a1ef6769db"
format_stamp: "Formatted at 2026-09-25 12:21:08 on dist-test-slave-clmx"
server_key: "46cb58707cff7d8430e2f8363361a48e"
server_key_iv: "f789ae48e3b09c868ab217b65b6d9992"
server_key_version: "encryptionkey@0"
I20260925 12:21:08.127903 21359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "06ecd32eec5c411f83e3c0a1ef6769db"
format_stamp: "Formatted at 2026-09-25 12:21:08 on dist-test-slave-clmx"
server_key: "46cb58707cff7d8430e2f8363361a48e"
server_key_iv: "f789ae48e3b09c868ab217b65b6d9992"
server_key_version: "encryptionkey@0"
I20260925 12:21:08.128989 21359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:21:08.129570 21374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:08.129719 21359 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:21:08.129791 21359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "06ecd32eec5c411f83e3c0a1ef6769db"
format_stamp: "Formatted at 2026-09-25 12:21:08 on dist-test-slave-clmx"
server_key: "46cb58707cff7d8430e2f8363361a48e"
server_key_iv: "f789ae48e3b09c868ab217b65b6d9992"
server_key_version: "encryptionkey@0"
I20260925 12:21:08.129837 21359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:08.151014 21359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:08.153182 21359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:08.153298 21359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:08.156831 21359 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46671
I20260925 12:21:08.156894 21426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46671 every 8 connection(s)
I20260925 12:21:08.157168 21359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:21:08.157918 21427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:08.159901 21427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db: Bootstrap starting.
I20260925 12:21:08.160436 21427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:08.160691 21427 log.cc:824] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:08.161325 21427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db: No bootstrap required, opened a new log
I20260925 12:21:08.162837 21427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ecd32eec5c411f83e3c0a1ef6769db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46671 } }
I20260925 12:21:08.162984 21427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:21:08.163028 21427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ecd32eec5c411f83e3c0a1ef6769db, State: Initialized, Role: FOLLOWER
I20260925 12:21:08.163188 21427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ecd32eec5c411f83e3c0a1ef6769db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46671 } }
I20260925 12:21:08.163283 21427 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:21:08.163323 21427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:21:08.163371 21427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:08.163965 21427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ecd32eec5c411f83e3c0a1ef6769db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46671 } }
I20260925 12:21:08.164088 21427 leader_election.cc:302] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06ecd32eec5c411f83e3c0a1ef6769db; no voters:
I20260925 12:21:08.164265 21427 leader_election.cc:288] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:21:08.164269 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 21359
I20260925 12:21:08.164305 21430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:21:08.164381 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:21:08.164417 21430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db [term 1 LEADER]: Becoming Leader. State: Replica: 06ecd32eec5c411f83e3c0a1ef6769db, State: Running, Role: LEADER
I20260925 12:21:08.164510 21427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:08.164491 21430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ecd32eec5c411f83e3c0a1ef6769db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46671 } }
I20260925 12:21:08.164559 392 external_mini_cluster.cc:1428] Setting key 6ce1725a56d557ae1ac8d21c194b8ea4
I20260925 12:21:08.164832 21430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06ecd32eec5c411f83e3c0a1ef6769db. Latest consensus state: current_term: 1 leader_uuid: "06ecd32eec5c411f83e3c0a1ef6769db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ecd32eec5c411f83e3c0a1ef6769db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46671 } } }
I20260925 12:21:08.164996 21430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:08.164994 21431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06ecd32eec5c411f83e3c0a1ef6769db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ecd32eec5c411f83e3c0a1ef6769db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46671 } } }
I20260925 12:21:08.165162 21431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:08.165331 21441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:21:08.166009 21441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:21:08.167562 21441 catalog_manager.cc:1347] Generated new cluster ID: 7d111342daad4f19b95eaf6df08a4ce1
I20260925 12:21:08.167686 21441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:21:08.170231 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:08.165107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:37467 (local address 127.0.0.1:46671)
0925 12:21:08.165274 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:08.165283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:08.165359 (+ 76us) server_negotiation.cc:407] Connection header received
0925 12:21:08.165541 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:08.165549 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:08.165604 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:08.165734 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:08.166091 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:08.167111 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:08.168256 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:08.168770 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:08.168859 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:08.169542 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:08.169545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:08.169545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:08.169749 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:08.169780 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:08.169791 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:08.169862 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:08.170026 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 12:21:08.170051 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260925 12:21:08.171095 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:21:08.176862 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:08.173515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:52805 (local address 127.0.0.1:46671)
0925 12:21:08.173543 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:08.173545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:08.173674 (+ 129us) server_negotiation.cc:407] Connection header received
0925 12:21:08.173728 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:08.173729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:08.173738 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:08.173789 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:08.173917 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:08.174637 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:08.175632 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:08.175926 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:08.175984 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:08.176617 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:08.176619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:08.176619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:08.176623 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:08.176636 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:08.176640 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:08.176697 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:08.176785 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 12:21:08.176796 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 12:21:08.179944 21441 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:21:08.180577 21441 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:21:08.187198 21441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06ecd32eec5c411f83e3c0a1ef6769db: Generated new TSK 0
I20260925 12:21:08.187345 21441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:21:08.192987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 21359
W20260925 12:21:08.196839 21344 connection.cc:535] client connection to 127.0.0.1:46671 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46671): Connection reset by peer (error 104)
2026-09-25T12:21:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1236 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:21:08 dist-test-slave-clmx krb5kdc[21458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:21:08 dist-test-slave-clmx krb5kdc[21458](info): set up 2 sockets
Sep 25 12:21:08 dist-test-slave-clmx krb5kdc[21458](info): commencing operation
krb5kdc: starting...
W20260925 12:21:09.171757 392 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:21:09 dist-test-slave-clmx krb5kdc[21458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:21:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:21:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:21:09.277688 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38197
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:21:09.351490 21471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:09.351727 21471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:09.351766 21471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:09.352819 21471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:09.352895 21471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:09.352916 21471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:09.352937 21471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:09.352962 21471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:09.354305 21471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:09.354559 21471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:09.354764 21471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:09.358958 21476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:09.358978 21477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:09.358979 21479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:09.359064 21471 server_base.cc:1034] running on GCE node
I20260925 12:21:09.359352 21471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:09.359606 21471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:09.360777 21471 hybrid_clock.cc:648] HybridClock initialized: now 1790338869360767 us; error 42 us; skew 500 ppm
Sep 25 12:21:09 dist-test-slave-clmx krb5kdc[21458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:09.363111 21471 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:21:09.363446 21471 webserver.cc:460] Webserver started at http://127.0.0.1:33013/ using document root <none> and password file <none>
I20260925 12:21:09.363601 21471 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:09.363698 21471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:09.363790 21471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:09.364704 21471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "b6e8216022a449a4a4355f78b4fd25fd"
format_stamp: "Formatted at 2026-09-25 12:21:09 on dist-test-slave-clmx"
server_key: "fc0bf2b67dc5d288af351b27bd6fca36"
server_key_iv: "4f8924a92943dfc3a026beaa8a530c56"
server_key_version: "encryptionkey@0"
I20260925 12:21:09.364998 21471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "b6e8216022a449a4a4355f78b4fd25fd"
format_stamp: "Formatted at 2026-09-25 12:21:09 on dist-test-slave-clmx"
server_key: "fc0bf2b67dc5d288af351b27bd6fca36"
server_key_iv: "4f8924a92943dfc3a026beaa8a530c56"
server_key_version: "encryptionkey@0"
I20260925 12:21:09.366114 21471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:21:09.366684 21486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:09.366837 21471 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:21:09.366917 21471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "b6e8216022a449a4a4355f78b4fd25fd"
format_stamp: "Formatted at 2026-09-25 12:21:09 on dist-test-slave-clmx"
server_key: "fc0bf2b67dc5d288af351b27bd6fca36"
server_key_iv: "4f8924a92943dfc3a026beaa8a530c56"
server_key_version: "encryptionkey@0"
I20260925 12:21:09.366986 21471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:09.369944 21471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:09.371686 21471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:09.371827 21471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:09.375280 21471 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38197
I20260925 12:21:09.375340 21538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38197 every 8 connection(s)
I20260925 12:21:09.375682 21471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:21:09.376577 21539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:09.378535 21539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd: Bootstrap starting.
I20260925 12:21:09.379081 21539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:09.379346 21539 log.cc:824] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:09.380040 21539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd: No bootstrap required, opened a new log
I20260925 12:21:09.381532 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 21471
I20260925 12:21:09.381650 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:21:09.381562 21539 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e8216022a449a4a4355f78b4fd25fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38197 } }
I20260925 12:21:09.381779 21539 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:21:09.381826 21539 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6e8216022a449a4a4355f78b4fd25fd, State: Initialized, Role: FOLLOWER
I20260925 12:21:09.381875 392 external_mini_cluster.cc:1428] Setting key d621d89c57eff8a2851f310d9745e01c
I20260925 12:21:09.381971 21539 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e8216022a449a4a4355f78b4fd25fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38197 } }
I20260925 12:21:09.382054 21539 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:21:09.382089 21539 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:21:09.382143 21539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:09.383106 21539 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e8216022a449a4a4355f78b4fd25fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38197 } }
I20260925 12:21:09.383240 21539 leader_election.cc:302] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6e8216022a449a4a4355f78b4fd25fd; no voters:
I20260925 12:21:09.383381 21539 leader_election.cc:288] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:21:09.383451 21544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:21:09.383596 21544 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd [term 1 LEADER]: Becoming Leader. State: Replica: b6e8216022a449a4a4355f78b4fd25fd, State: Running, Role: LEADER
I20260925 12:21:09.383610 21539 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:09.383742 21544 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e8216022a449a4a4355f78b4fd25fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38197 } }
I20260925 12:21:09.384033 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:09.382288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:57257 (local address 127.0.0.1:38197)
0925 12:21:09.382526 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:09.382534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:09.382565 (+ 31us) server_negotiation.cc:407] Connection header received
0925 12:21:09.382690 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:09.382698 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:09.382729 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:09.382880 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:09.383479 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:09.383482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:09.383483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:09.383487 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:09.383526 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:09.383539 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:09.383590 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:09.383806 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 12:21:09.383849 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":114,"threads_started":1}
I20260925 12:21:09.384068 21544 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6e8216022a449a4a4355f78b4fd25fd. Latest consensus state: current_term: 1 leader_uuid: "b6e8216022a449a4a4355f78b4fd25fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e8216022a449a4a4355f78b4fd25fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38197 } } }
I20260925 12:21:09.384323 21544 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:09.384181 21545 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6e8216022a449a4a4355f78b4fd25fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e8216022a449a4a4355f78b4fd25fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38197 } } }
I20260925 12:21:09.384570 21545 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:09.384827 21553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:21:09.385257 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:21:09.385286 21553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:21:09.386806 21553 catalog_manager.cc:1347] Generated new cluster ID: 0c31172b984b46879e44983aa667c513
I20260925 12:21:09.386873 21553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:21:09.388772 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:09.386693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:40305 (local address 127.0.0.1:38197)
0925 12:21:09.386726 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:09.386728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:09.387848 (+ 1120us) server_negotiation.cc:407] Connection header received
0925 12:21:09.387915 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:09.387916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:09.387921 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:09.387996 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:09.388537 (+ 541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:09.388538 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:09.388538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:09.388542 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:09.388554 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:09.388557 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:09.388613 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:09.388667 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 12:21:09.388681 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:21:09.395233 21553 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:21:09.395370 21553 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:21:09.397789 21553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6e8216022a449a4a4355f78b4fd25fd: Generated new TSK 0
I20260925 12:21:09.397948 21553 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:21:09.402024 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 21471
2026-09-25T12:21:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1209 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:21:09 dist-test-slave-clmx krb5kdc[21570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:21:09 dist-test-slave-clmx krb5kdc[21570](info): set up 2 sockets
Sep 25 12:21:09 dist-test-slave-clmx krb5kdc[21570](info): commencing operation
krb5kdc: starting...
W20260925 12:21:10.399586 392 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:21:10 dist-test-slave-clmx krb5kdc[21570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:21:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:21:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:21:10.504647 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33631
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:21:10.580317 21583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:10.580518 21583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:10.580550 21583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:10.581671 21583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:10.581736 21583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:10.581760 21583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:10.581786 21583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:10.581813 21583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:10.583168 21583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33631
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:10.583415 21583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:10.583642 21583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:10.588095 21588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:10.588068 21589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:10.588055 21591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:10.588048 21583 server_base.cc:1034] running on GCE node
I20260925 12:21:10.588510 21583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:10.588708 21583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:10.589860 21583 hybrid_clock.cc:648] HybridClock initialized: now 1790338870589848 us; error 29 us; skew 500 ppm
Sep 25 12:21:10 dist-test-slave-clmx krb5kdc[21570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:10.592206 21583 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:21:10.592646 21583 webserver.cc:460] Webserver started at http://127.0.0.1:37729/ using document root <none> and password file <none>
I20260925 12:21:10.592830 21583 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:10.592886 21583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:10.593025 21583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:10.593873 21583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "4539a5c51fbe4bec90b77d6afa3184f4"
format_stamp: "Formatted at 2026-09-25 12:21:10 on dist-test-slave-clmx"
server_key: "4a255c94121a6ac400c65c75f78f161e"
server_key_iv: "bc1b4044881570a91bbcaf4b16a3085f"
server_key_version: "encryptionkey@0"
I20260925 12:21:10.594197 21583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "4539a5c51fbe4bec90b77d6afa3184f4"
format_stamp: "Formatted at 2026-09-25 12:21:10 on dist-test-slave-clmx"
server_key: "4a255c94121a6ac400c65c75f78f161e"
server_key_iv: "bc1b4044881570a91bbcaf4b16a3085f"
server_key_version: "encryptionkey@0"
I20260925 12:21:10.595328 21583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 12:21:10.596004 21598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:10.596153 21583 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:10.596242 21583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "4539a5c51fbe4bec90b77d6afa3184f4"
format_stamp: "Formatted at 2026-09-25 12:21:10 on dist-test-slave-clmx"
server_key: "4a255c94121a6ac400c65c75f78f161e"
server_key_iv: "bc1b4044881570a91bbcaf4b16a3085f"
server_key_version: "encryptionkey@0"
I20260925 12:21:10.596310 21583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:10.611061 21583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:10.613194 21583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:10.613341 21583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:10.617026 21583 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33631
I20260925 12:21:10.617053 21650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33631 every 8 connection(s)
I20260925 12:21:10.617364 21583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:21:10.618090 21651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:10.618660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 21583
I20260925 12:21:10.618758 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:21:10.618937 392 external_mini_cluster.cc:1428] Setting key 600f76be383040ee2aec765fdda53c34
I20260925 12:21:10.620992 21651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4: Bootstrap starting.
I20260925 12:21:10.621577 21651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:10.621891 21651 log.cc:824] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:10.622574 21651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4: No bootstrap required, opened a new log
I20260925 12:21:10.625185 21651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4539a5c51fbe4bec90b77d6afa3184f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33631 } }
I20260925 12:21:10.625324 21651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:21:10.625342 21651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4539a5c51fbe4bec90b77d6afa3184f4, State: Initialized, Role: FOLLOWER
I20260925 12:21:10.625453 21651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4539a5c51fbe4bec90b77d6afa3184f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33631 } }
I20260925 12:21:10.625524 21651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 25 12:21:10 dist-test-slave-clmx krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 12:21:10.625545 21651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:21:10.625603 21651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:10.626128 21651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4539a5c51fbe4bec90b77d6afa3184f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33631 } }
I20260925 12:21:10.626240 21651 leader_election.cc:302] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4539a5c51fbe4bec90b77d6afa3184f4; no voters:
I20260925 12:21:10.626387 21651 leader_election.cc:288] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:21:10.626544 21656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:21:10.626652 21651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:10.626708 21656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4 [term 1 LEADER]: Becoming Leader. State: Replica: 4539a5c51fbe4bec90b77d6afa3184f4, State: Running, Role: LEADER
I20260925 12:21:10.627060 21656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4539a5c51fbe4bec90b77d6afa3184f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33631 } }
I20260925 12:21:10.628078 21657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4539a5c51fbe4bec90b77d6afa3184f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4539a5c51fbe4bec90b77d6afa3184f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33631 } } }
I20260925 12:21:10.628120 21658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4539a5c51fbe4bec90b77d6afa3184f4. Latest consensus state: current_term: 1 leader_uuid: "4539a5c51fbe4bec90b77d6afa3184f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4539a5c51fbe4bec90b77d6afa3184f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33631 } } }
I20260925 12:21:10.628268 21657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4 [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:10.628276 21658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4 [sys.catalog]: This master's current role is: LEADER
W20260925 12:21:10.628425 21671 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:21:10.628468 21671 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:21:10.628531 21673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:21:10.628655 21673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:21:10.629513 21654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:10.619397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:36757 (local address 127.0.0.1:33631)
0925 12:21:10.619553 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:10.619561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:10.619590 (+ 29us) server_negotiation.cc:407] Connection header received
0925 12:21:10.619989 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:10.619994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:10.620037 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:10.620155 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:10.620835 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:10.621770 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:10.622812 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:10.623200 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:10.623277 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:10.626023 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:10.626027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:10.626027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:10.626038 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:10.627897 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:10.628353 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:10.628355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:10.628356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:10.628443 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:10.628642 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:10.628642 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:10.628643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:10.628920 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:10.629056 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:10.629240 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 12:21:10.629330 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260925 12:21:10.629937 21673 catalog_manager.cc:1347] Generated new cluster ID: ad67875c1a9d4876a2a512b0a83368a2
I20260925 12:21:10.629989 21673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:21:10.630375 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:21:10.638484 21654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:10.631653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:56157 (local address 127.0.0.1:33631)
0925 12:21:10.631685 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:10.631687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:10.631845 (+ 158us) server_negotiation.cc:407] Connection header received
0925 12:21:10.631874 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:10.631875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:10.631885 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:10.631953 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:10.632485 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:10.633495 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:10.634377 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:10.634704 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:10.634776 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:10.635967 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:10.635969 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:10.635969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:10.635975 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:10.637415 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:10.637766 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:10.637767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:10.637767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:10.637821 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:10.638016 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:10.638016 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:10.638017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:10.638129 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:10.638212 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:10.638345 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 12:21:10.638401 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 12:21:10.649251 21673 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:21:10.649741 21673 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:21:10.651803 21673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4539a5c51fbe4bec90b77d6afa3184f4: Generated new TSK 0
I20260925 12:21:10.651970 21673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:21:10.655905 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 21583
W20260925 12:21:10.659891 21568 connection.cc:535] client connection to 127.0.0.1:33631 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33631): Connection reset by peer (error 104)
2026-09-25T12:21:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1253 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:21:10 dist-test-slave-clmx krb5kdc[21682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:21:10 dist-test-slave-clmx krb5kdc[21682](info): set up 2 sockets
Sep 25 12:21:10 dist-test-slave-clmx krb5kdc[21682](info): commencing operation
krb5kdc: starting...
W20260925 12:21:11.655903 392 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:21:11 dist-test-slave-clmx krb5kdc[21682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:21:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:21:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:21:11.762533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:21:11.837773 21695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:11.837936 21695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:11.837956 21695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:11.839046 21695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:11.839085 21695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:11.839099 21695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:11.839111 21695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:11.839161 21695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:11.840528 21695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36135
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:11.840749 21695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:11.840960 21695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:11.845368 21703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:11.845394 21701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:11.845386 21700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:11.845657 21695 server_base.cc:1034] running on GCE node
I20260925 12:21:11.845836 21695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:11.846088 21695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:11.847224 21695 hybrid_clock.cc:648] HybridClock initialized: now 1790338871847215 us; error 26 us; skew 500 ppm
Sep 25 12:21:11 dist-test-slave-clmx krb5kdc[21682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:11.849581 21695 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:21:11.849915 21695 webserver.cc:460] Webserver started at http://127.0.0.1:34475/ using document root <none> and password file <none>
I20260925 12:21:11.850088 21695 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:11.850126 21695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:11.850203 21695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:11.851073 21695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "668a9cf30a3142aa9284608ecdeb22e0"
format_stamp: "Formatted at 2026-09-25 12:21:11 on dist-test-slave-clmx"
server_key: "7dbfc02a2b5ee6743875f279b558705f"
server_key_iv: "14cbf8512543d994106944b100b251a5"
server_key_version: "encryptionkey@0"
I20260925 12:21:11.851341 21695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "668a9cf30a3142aa9284608ecdeb22e0"
format_stamp: "Formatted at 2026-09-25 12:21:11 on dist-test-slave-clmx"
server_key: "7dbfc02a2b5ee6743875f279b558705f"
server_key_iv: "14cbf8512543d994106944b100b251a5"
server_key_version: "encryptionkey@0"
I20260925 12:21:11.852597 21695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:21:11.853233 21710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:11.853374 21695 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:21:11.853451 21695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "668a9cf30a3142aa9284608ecdeb22e0"
format_stamp: "Formatted at 2026-09-25 12:21:11 on dist-test-slave-clmx"
server_key: "7dbfc02a2b5ee6743875f279b558705f"
server_key_iv: "14cbf8512543d994106944b100b251a5"
server_key_version: "encryptionkey@0"
I20260925 12:21:11.853502 21695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:11.874833 21695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:11.876480 21695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:11.876579 21695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:11.880092 21695 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36135
I20260925 12:21:11.880120 21762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36135 every 8 connection(s)
I20260925 12:21:11.880443 21695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:21:11.881356 21763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:11.883469 21763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0: Bootstrap starting.
I20260925 12:21:11.884052 21763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:11.884343 21763 log.cc:824] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:11.885030 21763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0: No bootstrap required, opened a new log
I20260925 12:21:11.886339 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 21695
I20260925 12:21:11.886421 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:21:11.886621 392 external_mini_cluster.cc:1428] Setting key 5795ea000174cc5e125fd8539f725a75
I20260925 12:21:11.886663 21763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668a9cf30a3142aa9284608ecdeb22e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36135 } }
I20260925 12:21:11.886855 21763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:21:11.886915 21763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668a9cf30a3142aa9284608ecdeb22e0, State: Initialized, Role: FOLLOWER
I20260925 12:21:11.887077 21763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668a9cf30a3142aa9284608ecdeb22e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36135 } }
I20260925 12:21:11.887189 21763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:21:11.887245 21763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:21:11.887298 21763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:11.888216 21763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668a9cf30a3142aa9284608ecdeb22e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36135 } }
I20260925 12:21:11.888329 21763 leader_election.cc:302] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 668a9cf30a3142aa9284608ecdeb22e0; no voters:
I20260925 12:21:11.888511 21763 leader_election.cc:288] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:21:11.888531 21768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:21:11.888633 21768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0 [term 1 LEADER]: Becoming Leader. State: Replica: 668a9cf30a3142aa9284608ecdeb22e0, State: Running, Role: LEADER
I20260925 12:21:11.888748 21763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:11.888756 21768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668a9cf30a3142aa9284608ecdeb22e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36135 } }
I20260925 12:21:11.889140 21769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "668a9cf30a3142aa9284608ecdeb22e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668a9cf30a3142aa9284608ecdeb22e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36135 } } }
I20260925 12:21:11.889159 21770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 668a9cf30a3142aa9284608ecdeb22e0. Latest consensus state: current_term: 1 leader_uuid: "668a9cf30a3142aa9284608ecdeb22e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668a9cf30a3142aa9284608ecdeb22e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36135 } } }
I20260925 12:21:11.889253 21769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0 [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:11.889274 21770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0 [sys.catalog]: This master's current role is: LEADER
W20260925 12:21:11.890475 21784 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:21:11.890515 21784 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:21:11.890862 21775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:21:11.891042 21775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:21:11.892009 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:11.887074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:34479 (local address 127.0.0.1:36135)
0925 12:21:11.887290 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:11.887298 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:11.887326 (+ 28us) server_negotiation.cc:407] Connection header received
0925 12:21:11.887436 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:11.887441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:11.887481 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:11.887615 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:11.887885 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:11.889047 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:11.890001 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:11.890580 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:11.890676 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:11.891344 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:11.891347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:11.891348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:11.891561 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:11.891595 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:11.891609 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:11.891715 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:11.891815 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 12:21:11.891840 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":90,"threads_started":1}
I20260925 12:21:11.892783 21775 catalog_manager.cc:1347] Generated new cluster ID: 3cd003aa3eca402b874c2a14987c2b4c
I20260925 12:21:11.892834 21775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:21:11.892906 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:21:11.897673 21775 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:21:11.898147 21775 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:21:11.898954 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:11.894214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:36675 (local address 127.0.0.1:36135)
0925 12:21:11.894242 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:11.894244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:11.895867 (+ 1623us) server_negotiation.cc:407] Connection header received
0925 12:21:11.895948 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:11.895948 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:11.895958 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:11.895983 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:11.896154 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:11.896834 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:11.897803 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:11.898085 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:11.898153 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:11.898730 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:11.898731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:11.898731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:11.898735 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:11.898747 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:11.898751 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:11.898815 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:11.898874 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 12:21:11.898892 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":14}
I20260925 12:21:11.902791 21775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 668a9cf30a3142aa9284608ecdeb22e0: Generated new TSK 0
I20260925 12:21:11.902931 21775 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:21:11.905403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 21695
W20260925 12:21:11.909253 21680 connection.cc:535] client connection to 127.0.0.1:36135 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36135): Connection reset by peer (error 104)
2026-09-25T12:21:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1249 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:21:11 dist-test-slave-clmx krb5kdc[21794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:21:11 dist-test-slave-clmx krb5kdc[21794](info): set up 2 sockets
Sep 25 12:21:11 dist-test-slave-clmx krb5kdc[21794](info): commencing operation
krb5kdc: starting...
W20260925 12:21:12.883859 392 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:21:12 dist-test-slave-clmx krb5kdc[21794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:21:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:21:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:21:12.988335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43995
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:21:13.062582 21807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:13.062785 21807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:13.062805 21807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:13.063830 21807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:13.063861 21807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:13.063874 21807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:13.063887 21807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:13.063899 21807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:13.065250 21807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43995
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:13.065441 21807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:13.065598 21807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:13.069840 21813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:13.069832 21812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:13.069868 21807 server_base.cc:1034] running on GCE node
W20260925 12:21:13.069844 21815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:13.070232 21807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:13.070436 21807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:13.071578 21807 hybrid_clock.cc:648] HybridClock initialized: now 1790338873071564 us; error 29 us; skew 500 ppm
Sep 25 12:21:13 dist-test-slave-clmx krb5kdc[21794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:13.073840 21807 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:21:13.074260 21807 webserver.cc:460] Webserver started at http://127.0.0.1:41095/ using document root <none> and password file <none>
I20260925 12:21:13.074474 21807 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:13.074535 21807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:13.074658 21807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:13.075610 21807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "5bac54d76e29479e9b70fc756e401a2a"
format_stamp: "Formatted at 2026-09-25 12:21:13 on dist-test-slave-clmx"
server_key: "340ea8f6985d474dc449b38868341fb0"
server_key_iv: "15ee5920fd484823bb405fe1319fc47c"
server_key_version: "encryptionkey@0"
I20260925 12:21:13.076007 21807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "5bac54d76e29479e9b70fc756e401a2a"
format_stamp: "Formatted at 2026-09-25 12:21:13 on dist-test-slave-clmx"
server_key: "340ea8f6985d474dc449b38868341fb0"
server_key_iv: "15ee5920fd484823bb405fe1319fc47c"
server_key_version: "encryptionkey@0"
I20260925 12:21:13.077124 21807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:21:13.077730 21822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:13.077872 21807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:21:13.077960 21807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "5bac54d76e29479e9b70fc756e401a2a"
format_stamp: "Formatted at 2026-09-25 12:21:13 on dist-test-slave-clmx"
server_key: "340ea8f6985d474dc449b38868341fb0"
server_key_iv: "15ee5920fd484823bb405fe1319fc47c"
server_key_version: "encryptionkey@0"
I20260925 12:21:13.078035 21807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:13.081025 21807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:13.082670 21807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:13.082834 21807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:13.086506 21807 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43995
I20260925 12:21:13.086532 21874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43995 every 8 connection(s)
I20260925 12:21:13.086798 21807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:21:13.087674 21875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:13.089485 21875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a: Bootstrap starting.
I20260925 12:21:13.089926 21875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:13.090127 21875 log.cc:824] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:13.090651 21875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a: No bootstrap required, opened a new log
I20260925 12:21:13.092018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 21807
I20260925 12:21:13.092145 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:21:13.092167 21875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bac54d76e29479e9b70fc756e401a2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43995 } }
I20260925 12:21:13.092329 21875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:21:13.092341 392 external_mini_cluster.cc:1428] Setting key 1e2482dcb2776d67ee6399a2421e359a
I20260925 12:21:13.092355 21875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bac54d76e29479e9b70fc756e401a2a, State: Initialized, Role: FOLLOWER
I20260925 12:21:13.092547 21875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bac54d76e29479e9b70fc756e401a2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43995 } }
I20260925 12:21:13.092635 21875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:21:13.092670 21875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:21:13.092720 21875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:13.093463 21875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bac54d76e29479e9b70fc756e401a2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43995 } }
I20260925 12:21:13.093596 21875 leader_election.cc:302] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bac54d76e29479e9b70fc756e401a2a; no voters:
I20260925 12:21:13.093751 21875 leader_election.cc:288] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:21:13.093799 21880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:21:13.093958 21880 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a [term 1 LEADER]: Becoming Leader. State: Replica: 5bac54d76e29479e9b70fc756e401a2a, State: Running, Role: LEADER
I20260925 12:21:13.093978 21875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:13.094094 21880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bac54d76e29479e9b70fc756e401a2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43995 } }
I20260925 12:21:13.094439 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:13.092880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:58569 (local address 127.0.0.1:43995)
0925 12:21:13.093035 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:13.093042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:13.093088 (+ 46us) server_negotiation.cc:407] Connection header received
0925 12:21:13.093175 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:13.093180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:13.093200 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:13.093326 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:13.093994 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:13.093998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:13.093998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:13.094003 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:13.094043 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:13.094057 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:13.094114 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:13.094238 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 12:21:13.094277 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260925 12:21:13.094602 21880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bac54d76e29479e9b70fc756e401a2a. Latest consensus state: current_term: 1 leader_uuid: "5bac54d76e29479e9b70fc756e401a2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bac54d76e29479e9b70fc756e401a2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43995 } } }
I20260925 12:21:13.094738 21880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:13.095248 21889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:21:13.095350 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:21:13.096200 21881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bac54d76e29479e9b70fc756e401a2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bac54d76e29479e9b70fc756e401a2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43995 } } }
I20260925 12:21:13.096295 21881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:13.096307 21889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:21:13.097721 21889 catalog_manager.cc:1347] Generated new cluster ID: 9cfb775ccb88402abe726ab26f5baeea
I20260925 12:21:13.097765 21889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:21:13.097821 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:13.096751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:44767 (local address 127.0.0.1:43995)
0925 12:21:13.096784 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:13.096786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:13.096883 (+ 97us) server_negotiation.cc:407] Connection header received
0925 12:21:13.096963 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:13.096964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:13.096968 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:13.097020 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:13.097547 (+ 527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:13.097548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:13.097548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:13.097552 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:13.097564 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:13.097568 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:13.097629 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:13.097761 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 12:21:13.097773 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:21:13.112468 21889 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:21:13.112618 21889 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:21:13.118031 21889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bac54d76e29479e9b70fc756e401a2a: Generated new TSK 0
I20260925 12:21:13.118181 21889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:21:13.121420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 21807
2026-09-25T12:21:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:21:13 dist-test-slave-clmx krb5kdc[21906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:21:13 dist-test-slave-clmx krb5kdc[21906](info): set up 2 sockets
Sep 25 12:21:13 dist-test-slave-clmx krb5kdc[21906](info): commencing operation
krb5kdc: starting...
W20260925 12:21:14.095573 392 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:21:14 dist-test-slave-clmx krb5kdc[21906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:21:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:21:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:21:14.200237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39427
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:21:14.273228 21919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:14.273420 21919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:14.273456 21919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:14.274511 21919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:14.274571 21919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:14.274593 21919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:14.274618 21919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:14.274641 21919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:14.276019 21919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39427
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:14.276289 21919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:14.276489 21919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:14.280685 21925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:14.280725 21924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:14.280689 21927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:14.280802 21919 server_base.cc:1034] running on GCE node
I20260925 12:21:14.281000 21919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:14.281243 21919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:14.282378 21919 hybrid_clock.cc:648] HybridClock initialized: now 1790338874282358 us; error 34 us; skew 500 ppm
Sep 25 12:21:14 dist-test-slave-clmx krb5kdc[21906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:14.284657 21919 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:21:14.285089 21919 webserver.cc:460] Webserver started at http://127.0.98.62:39655/ using document root <none> and password file <none>
I20260925 12:21:14.285300 21919 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:14.285341 21919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:14.285477 21919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:14.286417 21919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "45af94af82f244bda0d03b60d3504dff"
format_stamp: "Formatted at 2026-09-25 12:21:14 on dist-test-slave-clmx"
server_key: "6d86e1dffd33b38f26603205bdcc2e58"
server_key_iv: "463ee915aa1d58dbefa3b666d5ece004"
server_key_version: "encryptionkey@0"
I20260925 12:21:14.286741 21919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "45af94af82f244bda0d03b60d3504dff"
format_stamp: "Formatted at 2026-09-25 12:21:14 on dist-test-slave-clmx"
server_key: "6d86e1dffd33b38f26603205bdcc2e58"
server_key_iv: "463ee915aa1d58dbefa3b666d5ece004"
server_key_version: "encryptionkey@0"
I20260925 12:21:14.288049 21919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:21:14.288710 21934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:14.288866 21919 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:21:14.288962 21919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "45af94af82f244bda0d03b60d3504dff"
format_stamp: "Formatted at 2026-09-25 12:21:14 on dist-test-slave-clmx"
server_key: "6d86e1dffd33b38f26603205bdcc2e58"
server_key_iv: "463ee915aa1d58dbefa3b666d5ece004"
server_key_version: "encryptionkey@0"
I20260925 12:21:14.289057 21919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:14.302907 21919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:14.305423 21919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:14.305583 21919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:14.310209 21919 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39427
I20260925 12:21:14.310241 21986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39427 every 8 connection(s)
I20260925 12:21:14.310599 21919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:21:14.311468 21987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:14.313757 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 21919
I20260925 12:21:14.313871 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:21:14.314033 392 external_mini_cluster.cc:1428] Setting key 47accbf5d71999a50c4a182f97e60472
I20260925 12:21:14.314042 21987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff: Bootstrap starting.
I20260925 12:21:14.314692 21987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:14.315004 21987 log.cc:824] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:14.315681 21987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff: No bootstrap required, opened a new log
I20260925 12:21:14.317447 21987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45af94af82f244bda0d03b60d3504dff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39427 } }
I20260925 12:21:14.317654 21987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:21:14.317694 21987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45af94af82f244bda0d03b60d3504dff, State: Initialized, Role: FOLLOWER
I20260925 12:21:14.317816 21987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45af94af82f244bda0d03b60d3504dff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39427 } }
I20260925 12:21:14.317879 21987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:21:14.317916 21987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:21:14.317960 21987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:14.318504 21987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45af94af82f244bda0d03b60d3504dff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39427 } }
I20260925 12:21:14.318585 21987 leader_election.cc:302] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45af94af82f244bda0d03b60d3504dff; no voters:
I20260925 12:21:14.318747 21987 leader_election.cc:288] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:21:14.318773 21992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:21:14.318881 21992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff [term 1 LEADER]: Becoming Leader. State: Replica: 45af94af82f244bda0d03b60d3504dff, State: Running, Role: LEADER
I20260925 12:21:14.318992 21987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:14.319007 21992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45af94af82f244bda0d03b60d3504dff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39427 } }
I20260925 12:21:14.319305 21992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45af94af82f244bda0d03b60d3504dff. Latest consensus state: current_term: 1 leader_uuid: "45af94af82f244bda0d03b60d3504dff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45af94af82f244bda0d03b60d3504dff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39427 } } }
I20260925 12:21:14.319402 21994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45af94af82f244bda0d03b60d3504dff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45af94af82f244bda0d03b60d3504dff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39427 } } }
I20260925 12:21:14.319435 21992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:14.319487 21994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff [sys.catalog]: This master's current role is: LEADER
W20260925 12:21:14.320760 22008 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:21:14.320813 22008 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:21:14.320858 22000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:21:14.321009 22000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 12:21:14 dist-test-slave-clmx krb5kdc[21906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:21:14.322484 22000 catalog_manager.cc:1347] Generated new cluster ID: b2e908fca1e74c75849e05d9904ce430
I20260925 12:21:14.322536 22000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:21:14.326630 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:14.314458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58513 (local address 127.0.98.62:39427)
0925 12:21:14.314667 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:14.314677 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:14.314708 (+ 31us) server_negotiation.cc:407] Connection header received
0925 12:21:14.315292 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:14.315298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:14.315341 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:14.315495 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:14.316112 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:14.318012 (+ 1900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:14.318875 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:14.319442 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:14.319541 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:14.322232 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:14.322235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:14.322236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:14.322245 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:14.325117 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:14.325532 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:14.325535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:14.325535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:14.325620 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:14.325873 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:14.325874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:14.325874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:14.326106 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:14.326272 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:14.326364 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 12:21:14.326455 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260925 12:21:14.327833 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:21:14.329313 22000 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:21:14.329748 22000 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:21:14.333170 22000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45af94af82f244bda0d03b60d3504dff: Generated new TSK 0
I20260925 12:21:14.333305 22000 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:21:14.335888 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:14.329089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42121 (local address 127.0.98.62:39427)
0925 12:21:14.329123 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:14.329125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:14.329178 (+ 53us) server_negotiation.cc:407] Connection header received
0925 12:21:14.329246 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:14.329247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:14.329254 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:14.329299 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:14.329941 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:14.330678 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:14.331752 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:14.332072 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:14.332111 (+ 39us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:14.333453 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:14.333454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:14.333454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:14.333459 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:14.334750 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:14.335138 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:14.335139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:14.335139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:14.335184 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:14.335393 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:14.335393 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:14.335394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:14.335476 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:14.335561 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:14.335741 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 12:21:14.335795 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:21:14.337592 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 21919
W20260925 12:21:14.341394 21904 connection.cc:535] client connection to 127.0.98.62:39427 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39427): Connection reset by peer (error 104)
2026-09-25T12:21:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1215 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:21:14 dist-test-slave-clmx krb5kdc[22018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:21:14 dist-test-slave-clmx krb5kdc[22018](info): set up 2 sockets
Sep 25 12:21:14 dist-test-slave-clmx krb5kdc[22018](info): commencing operation
krb5kdc: starting...
W20260925 12:21:15.307727 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:21:15 dist-test-slave-clmx krb5kdc[22018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:21:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:21:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:21:15.414207 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33873
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:21:15.491565 22031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:15.491824 22031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:15.491846 22031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:15.492970 22031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:15.493017 22031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:15.493031 22031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:15.493041 22031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:15.493052 22031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:15.494342 22031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33873
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:15.494500 22031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:15.494701 22031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:15.498833 22037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:15.498872 22031 server_base.cc:1034] running on GCE node
W20260925 12:21:15.498847 22036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:15.498864 22039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:15.499271 22031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:15.499532 22031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:15.500681 22031 hybrid_clock.cc:648] HybridClock initialized: now 1790338875500687 us; error 45 us; skew 500 ppm
Sep 25 12:21:15 dist-test-slave-clmx krb5kdc[22018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:15.502837 22031 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:21:15.503259 22031 webserver.cc:460] Webserver started at http://127.0.98.62:35085/ using document root <none> and password file <none>
I20260925 12:21:15.503476 22031 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:15.503547 22031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:15.503687 22031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:15.504659 22031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "e90bfd43828e44ad953d7728c03fb274"
format_stamp: "Formatted at 2026-09-25 12:21:15 on dist-test-slave-clmx"
server_key: "64cd197cadce955ddebd6ce2fbb7f504"
server_key_iv: "f921b46857ea1de04e178de9f0351d9d"
server_key_version: "encryptionkey@0"
I20260925 12:21:15.505000 22031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "e90bfd43828e44ad953d7728c03fb274"
format_stamp: "Formatted at 2026-09-25 12:21:15 on dist-test-slave-clmx"
server_key: "64cd197cadce955ddebd6ce2fbb7f504"
server_key_iv: "f921b46857ea1de04e178de9f0351d9d"
server_key_version: "encryptionkey@0"
I20260925 12:21:15.506207 22031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:21:15.506860 22046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:15.507011 22031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:21:15.507104 22031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "e90bfd43828e44ad953d7728c03fb274"
format_stamp: "Formatted at 2026-09-25 12:21:15 on dist-test-slave-clmx"
server_key: "64cd197cadce955ddebd6ce2fbb7f504"
server_key_iv: "f921b46857ea1de04e178de9f0351d9d"
server_key_version: "encryptionkey@0"
I20260925 12:21:15.507180 22031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:15.525244 22031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:15.526782 22031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:15.526929 22031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:15.530522 22031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33873
I20260925 12:21:15.530575 22098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33873 every 8 connection(s)
I20260925 12:21:15.530892 22031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:21:15.531571 22099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:15.533906 22099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274: Bootstrap starting.
I20260925 12:21:15.534585 22099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:15.534896 22099 log.cc:824] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:15.535691 22099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274: No bootstrap required, opened a new log
I20260925 12:21:15.537518 22099 raft_consensus.cc:348] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90bfd43828e44ad953d7728c03fb274" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33873 } }
I20260925 12:21:15.537660 22099 raft_consensus.cc:374] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:21:15.537683 22099 raft_consensus.cc:729] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90bfd43828e44ad953d7728c03fb274, State: Initialized, Role: FOLLOWER
I20260925 12:21:15.537817 22099 consensus_queue.cc:260] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90bfd43828e44ad953d7728c03fb274" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33873 } }
I20260925 12:21:15.537909 22099 raft_consensus.cc:388] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:21:15.537948 22099 raft_consensus.cc:482] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:21:15.537992 22099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:15.538061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 22031
I20260925 12:21:15.538190 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:21:15.538367 392 external_mini_cluster.cc:1428] Setting key 4ee7335687e4bf77f49746c8d19ddf2e
I20260925 12:21:15.538589 22099 raft_consensus.cc:504] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90bfd43828e44ad953d7728c03fb274" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33873 } }
I20260925 12:21:15.538699 22099 leader_election.cc:302] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e90bfd43828e44ad953d7728c03fb274; no voters:
I20260925 12:21:15.538846 22099 leader_election.cc:288] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:21:15.538877 22103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:21:15.539047 22103 raft_consensus.cc:686] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274 [term 1 LEADER]: Becoming Leader. State: Replica: e90bfd43828e44ad953d7728c03fb274, State: Running, Role: LEADER
I20260925 12:21:15.539093 22099 sys_catalog.cc:564] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:15.539150 22103 consensus_queue.cc:237] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90bfd43828e44ad953d7728c03fb274" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33873 } }
I20260925 12:21:15.539433 22103 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e90bfd43828e44ad953d7728c03fb274. Latest consensus state: current_term: 1 leader_uuid: "e90bfd43828e44ad953d7728c03fb274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90bfd43828e44ad953d7728c03fb274" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33873 } } }
I20260925 12:21:15.539522 22103 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274 [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:15.539587 22106 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e90bfd43828e44ad953d7728c03fb274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90bfd43828e44ad953d7728c03fb274" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33873 } } }
I20260925 12:21:15.539676 22106 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274 [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:15.539785 22110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:21:15.540199 22110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:21:15.544734 22110 catalog_manager.cc:1347] Generated new cluster ID: de33700934bc455b8345019b246c1e2d
I20260925 12:21:15.544744 22105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:15.538944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54891 (local address 127.0.98.62:33873)
0925 12:21:15.540503 (+ 1559us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:15.540513 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:15.540546 (+ 33us) server_negotiation.cc:407] Connection header received
0925 12:21:15.540734 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:15.540738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:15.540775 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:15.540906 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:15.541102 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:15.541939 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:15.542738 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:15.543167 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:15.543300 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:15.544042 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:15.544045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:15.544045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:15.544224 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:15.544276 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:15.544290 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:15.544415 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:15.544494 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 12:21:15.544520 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1505,"thread_start_us":64,"threads_started":1}
I20260925 12:21:15.544796 22110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:21:15.545578 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:21:15.549328 22105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:15.546795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44945 (local address 127.0.98.62:33873)
0925 12:21:15.546842 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:15.546844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:15.546951 (+ 107us) server_negotiation.cc:407] Connection header received
0925 12:21:15.546985 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:15.546986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:15.546992 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:15.547034 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:15.547143 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:15.547545 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:15.548407 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:15.548575 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:15.548620 (+ 45us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:15.549090 (+ 470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:15.549090 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:15.549091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:15.549094 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:15.549124 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:15.549127 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:15.549167 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:15.549232 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 12:21:15.549239 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 12:21:15.555751 22110 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:21:15.556195 22110 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:21:15.577416 22110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e90bfd43828e44ad953d7728c03fb274: Generated new TSK 0
I20260925 12:21:15.577585 22110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:21:15.585981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 22031
W20260925 12:21:15.589712 22016 connection.cc:535] client connection to 127.0.98.62:33873 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:33873): Connection reset by peer (error 104)
2026-09-25T12:21:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1248 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:21:15 dist-test-slave-clmx krb5kdc[22130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:21:15 dist-test-slave-clmx krb5kdc[22130](info): set up 2 sockets
Sep 25 12:21:15 dist-test-slave-clmx krb5kdc[22130](info): commencing operation
krb5kdc: starting...
W20260925 12:21:16.563570 392 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:21:16 dist-test-slave-clmx krb5kdc[22130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:21:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:21:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:21:16.668443 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43127
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:21:16.742082 22143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:16.742256 22143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:16.742278 22143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:16.743310 22143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:16.743341 22143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:16.743355 22143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:16.743367 22143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:16.743379 22143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:16.744736 22143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43127
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:16.744899 22143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:16.745054 22143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:16.749343 22148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:16.749461 22143 server_base.cc:1034] running on GCE node
W20260925 12:21:16.749333 22149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:16.749357 22151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:16.749826 22143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:16.750131 22143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:16.751281 22143 hybrid_clock.cc:648] HybridClock initialized: now 1790338876751265 us; error 33 us; skew 500 ppm
Sep 25 12:21:16 dist-test-slave-clmx krb5kdc[22130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:16.758152 22143 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:21:16.758518 22143 webserver.cc:460] Webserver started at http://127.0.98.62:34837/ using document root <none> and password file <none>
I20260925 12:21:16.758709 22143 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:16.758769 22143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:16.758925 22143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:16.759932 22143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "7dc5f7f716794d7f99709925c1ef35eb"
format_stamp: "Formatted at 2026-09-25 12:21:16 on dist-test-slave-clmx"
server_key: "97b899af24d3e4300487115ad32adc81"
server_key_iv: "059d82a44cc828d1fdff04669fb2c1d3"
server_key_version: "encryptionkey@0"
I20260925 12:21:16.760265 22143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "7dc5f7f716794d7f99709925c1ef35eb"
format_stamp: "Formatted at 2026-09-25 12:21:16 on dist-test-slave-clmx"
server_key: "97b899af24d3e4300487115ad32adc81"
server_key_iv: "059d82a44cc828d1fdff04669fb2c1d3"
server_key_version: "encryptionkey@0"
I20260925 12:21:16.761482 22143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:21:16.762238 22158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:16.762404 22143 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:21:16.762499 22143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "7dc5f7f716794d7f99709925c1ef35eb"
format_stamp: "Formatted at 2026-09-25 12:21:16 on dist-test-slave-clmx"
server_key: "97b899af24d3e4300487115ad32adc81"
server_key_iv: "059d82a44cc828d1fdff04669fb2c1d3"
server_key_version: "encryptionkey@0"
I20260925 12:21:16.762579 22143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:16.765746 22143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:16.767947 22143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:16.768098 22143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:16.771813 22143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43127
I20260925 12:21:16.771840 22210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43127 every 8 connection(s)
I20260925 12:21:16.772212 22143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:21:16.773085 22211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:16.775080 22211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb: Bootstrap starting.
I20260925 12:21:16.775552 22211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:16.775838 22211 log.cc:824] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:16.776417 22211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb: No bootstrap required, opened a new log
I20260925 12:21:16.777801 22211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc5f7f716794d7f99709925c1ef35eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43127 } }
I20260925 12:21:16.777920 22211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:21:16.777941 22211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dc5f7f716794d7f99709925c1ef35eb, State: Initialized, Role: FOLLOWER
I20260925 12:21:16.778076 22211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc5f7f716794d7f99709925c1ef35eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43127 } }
I20260925 12:21:16.778149 22211 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:21:16.778168 22211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:21:16.778193 22211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:16.778609 22211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc5f7f716794d7f99709925c1ef35eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43127 } }
I20260925 12:21:16.778680 22211 leader_election.cc:302] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dc5f7f716794d7f99709925c1ef35eb; no voters:
I20260925 12:21:16.778810 22211 leader_election.cc:288] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:21:16.778837 22214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:21:16.778968 22214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb [term 1 LEADER]: Becoming Leader. State: Replica: 7dc5f7f716794d7f99709925c1ef35eb, State: Running, Role: LEADER
I20260925 12:21:16.779069 22211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:16.779121 22214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc5f7f716794d7f99709925c1ef35eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43127 } }
I20260925 12:21:16.779492 22216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dc5f7f716794d7f99709925c1ef35eb. Latest consensus state: current_term: 1 leader_uuid: "7dc5f7f716794d7f99709925c1ef35eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc5f7f716794d7f99709925c1ef35eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43127 } } }
I20260925 12:21:16.779524 22215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dc5f7f716794d7f99709925c1ef35eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc5f7f716794d7f99709925c1ef35eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43127 } } }
I20260925 12:21:16.779640 22216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:16.779654 22215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:16.780125 22218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:21:16.780623 22218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:21:16.781951 22218 catalog_manager.cc:1347] Generated new cluster ID: fe97ba3afc3a47dba2ff43416ad59497
I20260925 12:21:16.781991 22218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:21:16.782413 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 22143
I20260925 12:21:16.782524 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:21:16.782711 392 external_mini_cluster.cc:1428] Setting key bd92b3850ef9ce1a2ead3b70f900f6ab
I20260925 12:21:16.788504 22234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:16.783173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56251 (local address 127.0.98.62:43127)
0925 12:21:16.783327 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:16.783336 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:16.783437 (+ 101us) server_negotiation.cc:407] Connection header received
0925 12:21:16.783579 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:16.783586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:16.783637 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:16.783759 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:16.788093 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:16.788096 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:16.788097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:16.788101 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:16.788138 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:16.788151 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:16.788216 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:16.788276 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 12:21:16.788313 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260925 12:21:16.789320 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:21:16.790333 22218 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:21:16.790486 22218 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:21:16.791671 22234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:16.790724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60331 (local address 127.0.98.62:43127)
0925 12:21:16.790796 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:16.790798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:16.790847 (+ 49us) server_negotiation.cc:407] Connection header received
0925 12:21:16.790908 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:16.790909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:16.790913 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:16.790972 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:16.791465 (+ 493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:16.791466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:16.791466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:16.791470 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:16.791482 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:16.791485 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:16.791545 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:16.791593 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 12:21:16.791604 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 12:21:16.796777 22218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dc5f7f716794d7f99709925c1ef35eb: Generated new TSK 0
I20260925 12:21:16.796939 22218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:21:16.799912 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 22143
2026-09-25T12:21:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:21:16 dist-test-slave-clmx krb5kdc[22242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:21:16 dist-test-slave-clmx krb5kdc[22242](info): set up 2 sockets
Sep 25 12:21:16 dist-test-slave-clmx krb5kdc[22242](info): commencing operation
krb5kdc: starting...
W20260925 12:21:17.771341 392 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:21:17 dist-test-slave-clmx krb5kdc[22242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:21:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:21:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:21:17.875533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44297
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:21:17.950891 22255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:17.951104 22255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:17.951139 22255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:17.952656 22255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:17.952718 22255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:17.952745 22255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:17.952778 22255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:17.952806 22255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:17.954160 22255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:17.954392 22255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:17.954598 22255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:17.958869 22263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:17.958869 22260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:17.959022 22255 server_base.cc:1034] running on GCE node
W20260925 12:21:17.959092 22261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:17.959312 22255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:17.959592 22255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:17.960764 22255 hybrid_clock.cc:648] HybridClock initialized: now 1790338877960754 us; error 25 us; skew 500 ppm
Sep 25 12:21:17 dist-test-slave-clmx krb5kdc[22242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:17.963061 22255 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:21:17.963449 22255 webserver.cc:460] Webserver started at http://127.0.98.62:35915/ using document root <none> and password file <none>
I20260925 12:21:17.963692 22255 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:17.963755 22255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:17.963888 22255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:17.964869 22255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "e75751730c064a4386541b46a151d9ab"
format_stamp: "Formatted at 2026-09-25 12:21:17 on dist-test-slave-clmx"
server_key: "71aaacd11fd252c6d5f595dd1c019a44"
server_key_iv: "44a0c9f996d9f66313ec1944fdacbad1"
server_key_version: "encryptionkey@0"
I20260925 12:21:17.965216 22255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "e75751730c064a4386541b46a151d9ab"
format_stamp: "Formatted at 2026-09-25 12:21:17 on dist-test-slave-clmx"
server_key: "71aaacd11fd252c6d5f595dd1c019a44"
server_key_iv: "44a0c9f996d9f66313ec1944fdacbad1"
server_key_version: "encryptionkey@0"
I20260925 12:21:17.966415 22255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:21:17.967090 22270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:17.967262 22255 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:21:17.967348 22255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "e75751730c064a4386541b46a151d9ab"
format_stamp: "Formatted at 2026-09-25 12:21:17 on dist-test-slave-clmx"
server_key: "71aaacd11fd252c6d5f595dd1c019a44"
server_key_iv: "44a0c9f996d9f66313ec1944fdacbad1"
server_key_version: "encryptionkey@0"
I20260925 12:21:17.967445 22255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:17.982838 22255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:17.984972 22255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:17.985124 22255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:17.988814 22255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44297
I20260925 12:21:17.988847 22322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44297 every 8 connection(s)
I20260925 12:21:17.989188 22255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:21:17.989943 22323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:17.991925 22323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab: Bootstrap starting.
I20260925 12:21:17.992419 22323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:17.992647 22323 log.cc:824] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:17.993232 22323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab: No bootstrap required, opened a new log
I20260925 12:21:17.994596 22323 raft_consensus.cc:348] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75751730c064a4386541b46a151d9ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } }
I20260925 12:21:17.994714 22323 raft_consensus.cc:374] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:21:17.994733 22323 raft_consensus.cc:729] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75751730c064a4386541b46a151d9ab, State: Initialized, Role: FOLLOWER
I20260925 12:21:17.994855 22323 consensus_queue.cc:260] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75751730c064a4386541b46a151d9ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } }
I20260925 12:21:17.994935 22323 raft_consensus.cc:388] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:21:17.994956 22323 raft_consensus.cc:482] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:21:17.994983 22323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:17.995410 22323 raft_consensus.cc:504] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75751730c064a4386541b46a151d9ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } }
I20260925 12:21:17.995481 22323 leader_election.cc:302] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e75751730c064a4386541b46a151d9ab; no voters:
I20260925 12:21:17.995589 22323 leader_election.cc:288] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:21:17.995664 22326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:21:17.995790 22326 raft_consensus.cc:686] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab [term 1 LEADER]: Becoming Leader. State: Replica: e75751730c064a4386541b46a151d9ab, State: Running, Role: LEADER
I20260925 12:21:17.995846 22323 sys_catalog.cc:564] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:17.995893 22326 consensus_queue.cc:237] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75751730c064a4386541b46a151d9ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } }
I20260925 12:21:17.996222 22328 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader e75751730c064a4386541b46a151d9ab. Latest consensus state: current_term: 1 leader_uuid: "e75751730c064a4386541b46a151d9ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75751730c064a4386541b46a151d9ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } } }
I20260925 12:21:17.996358 22327 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e75751730c064a4386541b46a151d9ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75751730c064a4386541b46a151d9ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } } }
I20260925 12:21:17.996373 22328 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:17.996441 22327 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:17.996845 22335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:21:17.997313 22335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:21:17.998726 22335 catalog_manager.cc:1347] Generated new cluster ID: e52c6618439343739189b7b4091d5765
I20260925 12:21:17.998780 22335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:21:17.999296 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 22255
I20260925 12:21:17.999411 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:21:17.999584 392 external_mini_cluster.cc:1428] Setting key 5b8086fb35f878ecffdfbff7362bb06e
Sep 25 12:21:18 dist-test-slave-clmx krb5kdc[22242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:21:18.009024 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:17.999964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49973 (local address 127.0.98.62:44297)
0925 12:21:18.000255 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:18.000261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:18.000281 (+ 20us) server_negotiation.cc:407] Connection header received
0925 12:21:18.000358 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:18.000363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:18.000397 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:18.000513 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:18.001105 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:18.001986 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:18.002760 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:18.003568 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:18.003708 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:18.005991 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:18.005995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:18.005995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:18.006005 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:18.007542 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:18.007914 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:18.007920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:18.007920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:18.007997 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:18.008211 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:18.008212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:18.008212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:18.008429 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:18.008580 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:18.008678 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 12:21:18.008826 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":66,"threads_started":1}
I20260925 12:21:18.009791 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:21:18.016834 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:18.011025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54261 (local address 127.0.98.62:44297)
0925 12:21:18.011066 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:18.011068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:18.011189 (+ 121us) server_negotiation.cc:407] Connection header received
0925 12:21:18.011253 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:18.011254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:18.011262 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:18.011305 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:18.011823 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:18.012309 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:18.012986 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:18.013204 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:18.013292 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:18.014337 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:18.014339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:18.014339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:18.014345 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:18.015708 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:18.016083 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:18.016084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:18.016084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:18.016135 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:18.016337 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:18.016338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:18.016339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:18.016443 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:18.016556 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:18.016664 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 12:21:18.016731 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 12:21:18.040555 22335 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:21:18.041162 22335 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:21:18.045833 22335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e75751730c064a4386541b46a151d9ab: Generated new TSK 0
I20260925 12:21:18.045993 22335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:21:18.054491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 22255
2026-09-25T12:21:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1254 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:21:18 dist-test-slave-clmx krb5kdc[22354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:21:18 dist-test-slave-clmx krb5kdc[22354](info): set up 2 sockets
Sep 25 12:21:18 dist-test-slave-clmx krb5kdc[22354](info): commencing operation
krb5kdc: starting...
W20260925 12:21:19.031463 392 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:21:19 dist-test-slave-clmx krb5kdc[22354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:21:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:21:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:21:19.137429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37055
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:21:19.210951 22367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:19.211120 22367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:19.211141 22367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:19.212258 22367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:19.212297 22367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:19.212311 22367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:19.212360 22367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:19.212387 22367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:19.213804 22367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:19.214042 22367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:19.214247 22367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:19.218652 22375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:19.218701 22372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:19.218645 22373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:19.218767 22367 server_base.cc:1034] running on GCE node
I20260925 12:21:19.218998 22367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:19.219229 22367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:19.220356 22367 hybrid_clock.cc:648] HybridClock initialized: now 1790338879220342 us; error 30 us; skew 500 ppm
Sep 25 12:21:19 dist-test-slave-clmx krb5kdc[22354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:19.222381 22367 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:21:19.222812 22367 webserver.cc:460] Webserver started at http://127.0.98.62:46175/ using document root <none> and password file <none>
I20260925 12:21:19.223021 22367 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:19.223088 22367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:19.223232 22367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:19.224241 22367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "508e8a885c37403aa42b6893b2f448a3"
format_stamp: "Formatted at 2026-09-25 12:21:19 on dist-test-slave-clmx"
server_key: "a31f1c1834a864148149b80694a03139"
server_key_iv: "14586051eea205068fffae0366452287"
server_key_version: "encryptionkey@0"
I20260925 12:21:19.224573 22367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "508e8a885c37403aa42b6893b2f448a3"
format_stamp: "Formatted at 2026-09-25 12:21:19 on dist-test-slave-clmx"
server_key: "a31f1c1834a864148149b80694a03139"
server_key_iv: "14586051eea205068fffae0366452287"
server_key_version: "encryptionkey@0"
I20260925 12:21:19.225754 22367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:21:19.226377 22382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:19.226540 22367 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:21:19.226661 22367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "508e8a885c37403aa42b6893b2f448a3"
format_stamp: "Formatted at 2026-09-25 12:21:19 on dist-test-slave-clmx"
server_key: "a31f1c1834a864148149b80694a03139"
server_key_iv: "14586051eea205068fffae0366452287"
server_key_version: "encryptionkey@0"
I20260925 12:21:19.226737 22367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:19.238189 22367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:19.239873 22367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:19.240015 22367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:19.243552 22367 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37055
I20260925 12:21:19.243584 22434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37055 every 8 connection(s)
I20260925 12:21:19.243935 22367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:21:19.244670 22435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:19.246562 22435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3: Bootstrap starting.
I20260925 12:21:19.247100 22435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:19.247350 22435 log.cc:824] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:19.247979 22435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3: No bootstrap required, opened a new log
I20260925 12:21:19.249413 22435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "508e8a885c37403aa42b6893b2f448a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } }
I20260925 12:21:19.249522 22435 raft_consensus.cc:374] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:21:19.249545 22435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 508e8a885c37403aa42b6893b2f448a3, State: Initialized, Role: FOLLOWER
I20260925 12:21:19.249631 22435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "508e8a885c37403aa42b6893b2f448a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } }
I20260925 12:21:19.249706 22435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:21:19.249733 22435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:21:19.249780 22435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:19.250236 22435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "508e8a885c37403aa42b6893b2f448a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } }
I20260925 12:21:19.250342 22435 leader_election.cc:302] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 508e8a885c37403aa42b6893b2f448a3; no voters:
I20260925 12:21:19.250496 22435 leader_election.cc:288] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:21:19.250533 22438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:21:19.250684 22438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3 [term 1 LEADER]: Becoming Leader. State: Replica: 508e8a885c37403aa42b6893b2f448a3, State: Running, Role: LEADER
I20260925 12:21:19.250718 22435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:19.250808 22438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "508e8a885c37403aa42b6893b2f448a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } }
I20260925 12:21:19.250990 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 22367
I20260925 12:21:19.251096 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:21:19.251317 392 external_mini_cluster.cc:1428] Setting key 893536321e824e3eab63922cbe8a1b13
I20260925 12:21:19.251389 22440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 508e8a885c37403aa42b6893b2f448a3. Latest consensus state: current_term: 1 leader_uuid: "508e8a885c37403aa42b6893b2f448a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "508e8a885c37403aa42b6893b2f448a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } }
I20260925 12:21:19.251452 22440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3 [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:19.251770 22439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "508e8a885c37403aa42b6893b2f448a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "508e8a885c37403aa42b6893b2f448a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } }
I20260925 12:21:19.251915 22439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3 [sys.catalog]: This master's current role is: LEADER
W20260925 12:21:19.252580 22454 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:21:19.252646 22454 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:21:19.252740 22447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:21:19.252955 22447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:21:19.254215 22447 catalog_manager.cc:1347] Generated new cluster ID: f26d312347d04737b7b875406f6e569b
I20260925 12:21:19.254261 22447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:21:19.257318 22455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:19.251942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44283 (local address 127.0.98.62:37055)
0925 12:21:19.252282 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:19.252291 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:19.252323 (+ 32us) server_negotiation.cc:407] Connection header received
0925 12:21:19.252433 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:19.252458 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:19.252532 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:19.252902 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:19.252985 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:19.254280 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:19.255331 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:19.255923 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:19.256291 (+ 368us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:19.256592 (+ 301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:19.256596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:19.256596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:19.256767 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:19.256797 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:19.256811 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:19.257069 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:19.257096 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 12:21:19.257147 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":67,"threads_started":1}
I20260925 12:21:19.258143 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:21:19.262451 22455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:19.259473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53369 (local address 127.0.98.62:37055)
0925 12:21:19.259499 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:19.259501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:19.259508 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:21:19.259587 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:19.259587 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:19.259597 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:19.259681 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:19.259780 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:19.260402 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:19.261264 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:19.261571 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:19.261639 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:19.262231 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:19.262232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:19.262233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:19.262237 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:19.262249 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:19.262253 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:19.262310 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:19.262364 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 12:21:19.262386 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260925 12:21:19.264003 22447 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:21:19.264505 22447 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:21:19.267596 22447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 508e8a885c37403aa42b6893b2f448a3: Generated new TSK 0
I20260925 12:21:19.267786 22447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:21:19.273942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 22367
2026-09-25T12:21:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1219 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:21:19 dist-test-slave-clmx krb5kdc[22466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:21:19 dist-test-slave-clmx krb5kdc[22466](info): set up 2 sockets
Sep 25 12:21:19 dist-test-slave-clmx krb5kdc[22466](info): commencing operation
krb5kdc: starting...
W20260925 12:21:20.255745 392 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:21:20 dist-test-slave-clmx krb5kdc[22466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:21:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:21:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:21:20.363515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35871
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:21:20.439055 22479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:20.439221 22479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:20.439241 22479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:20.440464 22479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:20.440511 22479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:20.440524 22479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:20.440536 22479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:20.440580 22479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:20.441977 22479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:20.442210 22479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:20.442423 22479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:20.446920 22484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:20.446946 22485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:20.446940 22487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:20.447008 22479 server_base.cc:1034] running on GCE node
I20260925 12:21:20.447297 22479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:20.447505 22479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:20.448642 22479 hybrid_clock.cc:648] HybridClock initialized: now 1790338880448628 us; error 27 us; skew 500 ppm
Sep 25 12:21:20 dist-test-slave-clmx krb5kdc[22466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:20.451023 22479 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:21:20.451424 22479 webserver.cc:460] Webserver started at http://127.0.98.62:36485/ using document root <none> and password file <none>
I20260925 12:21:20.451682 22479 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:20.451726 22479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:20.451857 22479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:20.452829 22479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "67bab89591c6466593957703ebfe2c98"
format_stamp: "Formatted at 2026-09-25 12:21:20 on dist-test-slave-clmx"
server_key: "354ae55ea45d5363d53c7cdcaae7eb00"
server_key_iv: "ed38788dd6f3254221178f212d919f59"
server_key_version: "encryptionkey@0"
I20260925 12:21:20.453171 22479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "67bab89591c6466593957703ebfe2c98"
format_stamp: "Formatted at 2026-09-25 12:21:20 on dist-test-slave-clmx"
server_key: "354ae55ea45d5363d53c7cdcaae7eb00"
server_key_iv: "ed38788dd6f3254221178f212d919f59"
server_key_version: "encryptionkey@0"
I20260925 12:21:20.454504 22479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:21:20.455252 22494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:20.455462 22479 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:21:20.455531 22479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "67bab89591c6466593957703ebfe2c98"
format_stamp: "Formatted at 2026-09-25 12:21:20 on dist-test-slave-clmx"
server_key: "354ae55ea45d5363d53c7cdcaae7eb00"
server_key_iv: "ed38788dd6f3254221178f212d919f59"
server_key_version: "encryptionkey@0"
I20260925 12:21:20.455664 22479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:20.458640 22479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:20.460333 22479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:20.460459 22479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:20.464061 22479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35871
I20260925 12:21:20.464107 22546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35871 every 8 connection(s)
I20260925 12:21:20.464438 22479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:21:20.465276 22547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:20.467164 22547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98: Bootstrap starting.
I20260925 12:21:20.467164 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 22479
I20260925 12:21:20.467298 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:21:20.467465 392 external_mini_cluster.cc:1428] Setting key 1f60cf748e777949ff1656f680cdc12a
I20260925 12:21:20.467871 22547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:20.468235 22547 log.cc:824] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:20.468923 22547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98: No bootstrap required, opened a new log
I20260925 12:21:20.469708 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:20.467923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32973 (local address 127.0.98.62:35871)
0925 12:21:20.468086 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:20.468093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:20.468115 (+ 22us) server_negotiation.cc:407] Connection header received
0925 12:21:20.468551 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:20.468557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:20.468580 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:20.468687 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:20.469334 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:20.469337 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:20.469338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:20.469341 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:20.469379 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:20.469395 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:20.469482 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:20.469523 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 12:21:20.469558 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260925 12:21:20.470607 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:21:20.471171 22547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bab89591c6466593957703ebfe2c98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35871 } }
I20260925 12:21:20.471316 22547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:21:20.471395 22547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67bab89591c6466593957703ebfe2c98, State: Initialized, Role: FOLLOWER
I20260925 12:21:20.471517 22547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bab89591c6466593957703ebfe2c98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35871 } }
I20260925 12:21:20.471635 22547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:21:20.471702 22547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:21:20.471760 22547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:20.472437 22547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bab89591c6466593957703ebfe2c98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35871 } }
I20260925 12:21:20.472563 22547 leader_election.cc:302] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67bab89591c6466593957703ebfe2c98; no voters:
I20260925 12:21:20.472739 22547 leader_election.cc:288] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:21:20.472826 22557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:21:20.472965 22547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:20.473145 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:20.472107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39627 (local address 127.0.98.62:35871)
0925 12:21:20.472141 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:20.472142 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:20.472237 (+ 95us) server_negotiation.cc:407] Connection header received
0925 12:21:20.472324 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:20.472325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:20.472328 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:20.472371 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:20.472863 (+ 492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:20.472864 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:20.472864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:21:20.472869 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:20.472882 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:21:20.472886 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:20.472944 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:20.473067 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 12:21:20.473081 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 12:21:20.473001 22557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98 [term 1 LEADER]: Becoming Leader. State: Replica: 67bab89591c6466593957703ebfe2c98, State: Running, Role: LEADER
I20260925 12:21:20.474247 22557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bab89591c6466593957703ebfe2c98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35871 } }
I20260925 12:21:20.474572 22557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67bab89591c6466593957703ebfe2c98. Latest consensus state: current_term: 1 leader_uuid: "67bab89591c6466593957703ebfe2c98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bab89591c6466593957703ebfe2c98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35871 } } }
I20260925 12:21:20.474699 22557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98 [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:20.474750 22559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67bab89591c6466593957703ebfe2c98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bab89591c6466593957703ebfe2c98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35871 } } }
I20260925 12:21:20.474838 22559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98 [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:20.474903 22572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:21:20.475414 22572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:21:20.476761 22572 catalog_manager.cc:1347] Generated new cluster ID: b70a9b9f99d949b2ac3da1b865486301
I20260925 12:21:20.476827 22572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:21:20.497968 22572 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:21:20.498123 22572 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:21:20.503818 22572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67bab89591c6466593957703ebfe2c98: Generated new TSK 0
I20260925 12:21:20.503984 22572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:21:20.512245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 22479
2026-09-25T12:21:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1238 ms)
[----------] 18 tests from AuthTokenIssuingTest (22342 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:21:20 dist-test-slave-clmx krb5kdc[22578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:21:20 dist-test-slave-clmx krb5kdc[22578](info): set up 2 sockets
Sep 25 12:21:20 dist-test-slave-clmx krb5kdc[22578](info): commencing operation
krb5kdc: starting...
W20260925 12:21:21.495801 392 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:21:21 dist-test-slave-clmx krb5kdc[22578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:21:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:21:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:21:21.597386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46121
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46121,127.0.98.61:42703,127.0.98.60:38271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:21:21.671041 22591 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 12:21:21.671244 22591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:21.671259 22591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:21.671277 22591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:21.672289 22591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:21.672325 22591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:21.672340 22591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:21.672353 22591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:21.672363 22591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:21.673732 22591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46121,127.0.98.61:42703,127.0.98.60:38271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:21.673883 22591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:21.674017 22591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:21.678032 22597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:21.678041 22596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:21.678143 22591 server_base.cc:1034] running on GCE node
W20260925 12:21:21.678045 22599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:21.678439 22591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:21.678679 22591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:21.679831 22591 hybrid_clock.cc:648] HybridClock initialized: now 1790338881679828 us; error 38 us; skew 500 ppm
Sep 25 12:21:21 dist-test-slave-clmx krb5kdc[22578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:21.681968 22591 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:21:21.682312 22591 webserver.cc:460] Webserver started at http://127.0.98.62:34729/ using document root <none> and password file <none>
I20260925 12:21:21.682488 22591 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:21.682525 22591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:21.682611 22591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:21.683487 22591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "ff32f4a568c747268682b1091c8fb16b"
format_stamp: "Formatted at 2026-09-25 12:21:21 on dist-test-slave-clmx"
server_key: "c967d17c2754826e96d80393a1f9ae14"
server_key_iv: "9edbfb475dfb07e40dec860b33e7310c"
server_key_version: "encryptionkey@0"
I20260925 12:21:21.683843 22591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "ff32f4a568c747268682b1091c8fb16b"
format_stamp: "Formatted at 2026-09-25 12:21:21 on dist-test-slave-clmx"
server_key: "c967d17c2754826e96d80393a1f9ae14"
server_key_iv: "9edbfb475dfb07e40dec860b33e7310c"
server_key_version: "encryptionkey@0"
I20260925 12:21:21.684966 22591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:21:21.685554 22606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:21.685694 22591 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:21:21.685773 22591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "ff32f4a568c747268682b1091c8fb16b"
format_stamp: "Formatted at 2026-09-25 12:21:21 on dist-test-slave-clmx"
server_key: "c967d17c2754826e96d80393a1f9ae14"
server_key_iv: "9edbfb475dfb07e40dec860b33e7310c"
server_key_version: "encryptionkey@0"
I20260925 12:21:21.685820 22591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:21.695132 22591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:21.696578 22591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:21.696662 22591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:21.699957 22591 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46121
I20260925 12:21:21.700006 22658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46121 every 8 connection(s)
I20260925 12:21:21.700273 22591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:21:21.701036 22659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:21.701261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 22591
I20260925 12:21:21.701349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:21:21.701502 392 external_mini_cluster.cc:1428] Setting key e34dfb560d7ea844bcf229b98bd3843e
I20260925 12:21:21.702450 22659 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Sep 25 12:21:21 dist-test-slave-clmx krb5kdc[22578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260925 12:21:21.712778 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.703239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.0.98.62:46121)
0925 12:21:21.703380 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:21.703388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:21.703481 (+ 93us) server_negotiation.cc:407] Connection header received
0925 12:21:21.704132 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:21.704135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:21.704156 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:21.704279 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:21.704759 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.705927 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.707268 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.707539 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.707610 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:21.709680 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:21.709686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:21.709687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:21.709692 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:21.711243 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:21.711680 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:21.711682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:21.711683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:21.711747 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:21.711963 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:21.711964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:21.711964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:21.712156 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:21.712238 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:21.712439 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 12:21:21.712558 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":55,"threads_started":1}
I20260925 12:21:21.712781 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.703168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46121 (local address 127.0.0.1:55350)
0925 12:21:21.703347 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:21.703385 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 12:21:21.703977 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:21:21.704293 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:21:21.704296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:21:21.704509 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:21:21.704666 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:21.704669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.706022 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.706023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:21.707184 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:21.707187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.707661 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.707663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:21.707745 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:21.708176 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:21:21.708185 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:21:21.709589 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:21:21.711359 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:21.711364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:21.711366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:21.711572 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:21.711814 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:21.711815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:21.711815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:21.711897 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:21.712230 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:21:21.712235 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:21:21.712289 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 12:21:21.712403 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 12:21:21.712625 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":92,"thread_start_us":94,"threads_started":1}
I20260925 12:21:21.713732 22659 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:21:21.714020 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.713932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42703 (local address 127.0.0.1:41056)
0925 12:21:21.713967 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:21.714002 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 12:21:21.714145 22607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42703: connect: Connection refused (error 111)
I20260925 12:21:21.714526 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.714443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42703 (local address 127.0.0.1:41062)
0925 12:21:21.714486 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:21.714507 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 12:21:21.714629 22659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42703: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42703: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 12:21:21.724627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42703
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46121,127.0.98.61:42703,127.0.98.60:38271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:21:21.763902 22659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } attempt: 1
I20260925 12:21:21.764428 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.764302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42703 (local address 127.0.0.1:41072)
0925 12:21:21.764363 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:21.764401 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 12:21:21.764784 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.764704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42703 (local address 127.0.0.1:41080)
0925 12:21:21.764746 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:21.764771 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 12:21:21.764899 22659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42703: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42703: connect: Connection refused (error 111)
W20260925 12:21:21.799093 22666 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 12:21:21.799265 22666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:21.799297 22666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:21.799314 22666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:21.800256 22666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:21.800287 22666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:21.800300 22666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:21.800311 22666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:21.800323 22666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:21.801542 22666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46121,127.0.98.61:42703,127.0.98.60:38271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:21.801692 22666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:21.801831 22666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:21.806213 22674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:21.806262 22671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:21.806393 22666 server_base.cc:1034] running on GCE node
W20260925 12:21:21.806443 22672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:21.806643 22666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:21.806874 22666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:21.808001 22666 hybrid_clock.cc:648] HybridClock initialized: now 1790338881807988 us; error 28 us; skew 500 ppm
Sep 25 12:21:21 dist-test-slave-clmx krb5kdc[22578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:21.816574 22666 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 12:21:21.816984 22666 webserver.cc:460] Webserver started at http://127.0.98.61:40575/ using document root <none> and password file <none>
I20260925 12:21:21.817199 22666 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:21.817239 22666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:21.817381 22666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:21.818332 22666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-1/data/instance:
uuid: "6068874d333e4b1da875b017fd80accd"
format_stamp: "Formatted at 2026-09-25 12:21:21 on dist-test-slave-clmx"
server_key: "b3761d76d62b651699ae0868a74a68be"
server_key_iv: "0e54772ebaf591f2caead0e05b3ec7b0"
server_key_version: "encryptionkey@0"
I20260925 12:21:21.818626 22666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-1/wal/instance:
uuid: "6068874d333e4b1da875b017fd80accd"
format_stamp: "Formatted at 2026-09-25 12:21:21 on dist-test-slave-clmx"
server_key: "b3761d76d62b651699ae0868a74a68be"
server_key_iv: "0e54772ebaf591f2caead0e05b3ec7b0"
server_key_version: "encryptionkey@0"
I20260925 12:21:21.819720 22666 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:21:21.820356 22681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:21.820528 22666 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:21.820592 22666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-1/wal
uuid: "6068874d333e4b1da875b017fd80accd"
format_stamp: "Formatted at 2026-09-25 12:21:21 on dist-test-slave-clmx"
server_key: "b3761d76d62b651699ae0868a74a68be"
server_key_iv: "0e54772ebaf591f2caead0e05b3ec7b0"
server_key_version: "encryptionkey@0"
I20260925 12:21:21.820693 22666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:21.833051 22659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } attempt: 2
I20260925 12:21:21.833390 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.833276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42703 (local address 127.0.0.1:41090)
0925 12:21:21.833331 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:21.833366 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 12:21:21.833719 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.833633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42703 (local address 127.0.0.1:41102)
0925 12:21:21.833674 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:21.833695 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 12:21:21.833801 22659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42703: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42703: connect: Connection refused (error 111)
I20260925 12:21:21.847692 22666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:21.849364 22666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:21.849527 22666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:21.853338 22666 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42703
I20260925 12:21:21.853370 22733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42703 every 8 connection(s)
I20260925 12:21:21.853688 22666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-1/data/info.pb
I20260925 12:21:21.854624 22734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:21.856015 22734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:21:21.858569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 22666
I20260925 12:21:21.858649 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-1/wal/instance
I20260925 12:21:21.858806 392 external_mini_cluster.cc:1428] Setting key 995c375cfc014f3cb38422428d604294
Sep 25 12:21:21 dist-test-slave-clmx krb5kdc[22578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:21:21.864768 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.856640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55366 (local address 127.0.98.62:46121)
0925 12:21:21.856695 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:21.856697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:21.856847 (+ 150us) server_negotiation.cc:407] Connection header received
0925 12:21:21.857257 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:21.857258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:21.857268 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:21.857320 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:21.857829 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.858278 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.859707 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.859917 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.859969 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:21.861940 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:21.861942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:21.861942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:21.861956 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:21.863320 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:21.863827 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:21.863829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:21.863829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:21.863883 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:21.864138 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:21.864139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:21.864139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:21.864276 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:21.864351 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:21.864600 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 12:21:21.864679 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 12:21:21.864953 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.856589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46121 (local address 127.0.0.1:55366)
0925 12:21:21.856729 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:21.856765 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:21:21.857180 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:21:21.857380 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:21:21.857383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:21:21.857591 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:21:21.857759 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:21.857762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.858350 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.858350 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:21.859580 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:21.859583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.860007 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.860008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:21.860176 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:21.860638 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:21:21.860645 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:21:21.861839 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:21:21.863439 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:21.863445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:21.863447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:21.863749 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:21.863943 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:21.863943 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:21.863944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:21.864071 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:21.864344 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:21:21.864350 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:21:21.864478 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 12:21:21.864581 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 12:21:21.864727 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260925 12:21:21.865540 22734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 12:21:21 dist-test-slave-clmx krb5kdc[22578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 12:21:21.872275 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.865802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42703 (local address 127.0.0.1:41112)
0925 12:21:21.865849 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:21.865860 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:21:21.865897 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:21:21.866158 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:21:21.866159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:21:21.866312 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:21:21.866353 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:21.866354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.867254 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.867255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:21.867870 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:21.867871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.868425 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.868427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:21.868485 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:21.868834 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:21:21.868839 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:21:21.869807 (+ 968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:21:21.871321 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:21.871322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:21.871322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:21.871471 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:21.871704 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:21.871704 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:21.871705 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:21.871752 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:21.871995 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:21:21.871996 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:21:21.872028 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 12:21:21.872098 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 12:21:21.872183 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 12:21:21.872406 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.865868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41112 (local address 127.0.98.61:42703)
0925 12:21:21.866007 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:21.866012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:21.866032 (+ 20us) server_negotiation.cc:407] Connection header received
0925 12:21:21.866051 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:21.866053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:21.866064 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:21.866157 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:21.866448 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.867156 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.867956 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.868349 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.868418 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:21.869890 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:21.869896 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:21.869897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:21.869902 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:21.871229 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:21.871556 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:21.871559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:21.871559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:21.871648 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:21.871815 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:21.871815 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:21.871816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:21.871932 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:21.872014 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:21.872125 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 12:21:21.872242 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260925 12:21:21.873135 22734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:21:21.873433 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.873329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38271 (local address 127.0.0.1:53814)
0925 12:21:21.873373 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:21.873418 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 12:21:21.873579 22682 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38271: connect: Connection refused (error 111)
I20260925 12:21:21.873963 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.873878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38271 (local address 127.0.0.1:53822)
0925 12:21:21.873923 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:21.873945 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 12:21:21.874061 22734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38271: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38271: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 12:21:21.880085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38271
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46121,127.0.98.61:42703,127.0.98.60:38271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:21:21.923257 22734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } attempt: 1
I20260925 12:21:21.923707 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.923572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38271 (local address 127.0.0.1:53834)
0925 12:21:21.923641 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:21.923681 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 12:21:21.924068 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.923986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38271 (local address 127.0.0.1:53846)
0925 12:21:21.924030 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:21.924053 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 12:21:21.924180 22734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38271: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38271: connect: Connection refused (error 111)
I20260925 12:21:21.924927 22659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } attempt: 3
Sep 25 12:21:21 dist-test-slave-clmx krb5kdc[22578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 12:21:21.931385 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.925228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41120 (local address 127.0.98.61:42703)
0925 12:21:21.925287 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:21.925289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:21.925299 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:21:21.925358 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:21.925360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:21.925369 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:21.925452 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:21.925760 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.926260 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.926946 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.927172 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.927259 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:21.928960 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:21.928963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:21.928963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:21.928974 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:21.930295 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:21.930639 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:21.930640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:21.930641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:21.930694 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:21.930882 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:21.930883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:21.930883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:21.930987 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:21.931059 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:21.931206 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 12:21:21.931276 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 12:21:21.931391 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.925191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42703 (local address 127.0.0.1:41120)
0925 12:21:21.925230 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:21.925239 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:21:21.925292 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:21:21.925455 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:21:21.925456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:21:21.925613 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:21:21.925671 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:21.925673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.926346 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.926347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:21.926856 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:21.926858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.927257 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.927257 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:21.927333 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:21.927778 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:21:21.927782 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:21:21.928869 (+ 1087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:21:21.930392 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:21.930393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:21.930393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:21.930557 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:21.930756 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:21.930757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:21.930757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:21.930811 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:21.931050 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:21:21.931051 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:21:21.931099 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 12:21:21.931183 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 12:21:21.931274 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 12:21:21.931746 22659 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:21:21.932057 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.931967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38271 (local address 127.0.0.1:53856)
0925 12:21:21.932015 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:21.932040 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 12:21:21.932338 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.932251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38271 (local address 127.0.0.1:53858)
0925 12:21:21.932298 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:21.932319 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 12:21:21.932444 22659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38271: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38271: connect: Connection refused (error 111)
W20260925 12:21:21.955536 22741 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 12:21:21.955785 22741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:21.955823 22741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:21.955865 22741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:21.956895 22741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:21.956950 22741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:21.956972 22741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:21.956995 22741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:21.957017 22741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:21.958279 22741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46121,127.0.98.61:42703,127.0.98.60:38271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:21.958525 22741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:21.958716 22741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:21.962898 22749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:21.962888 22747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:21.963049 22741 server_base.cc:1034] running on GCE node
W20260925 12:21:21.962955 22746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:21.963304 22741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:21.963580 22659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } attempt: 1
I20260925 12:21:21.963680 22741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:21.963943 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.963822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38271 (local address 127.0.0.1:53864)
0925 12:21:21.963872 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:21.963905 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 12:21:21.964263 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.964146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38271 (local address 127.0.0.1:53866)
0925 12:21:21.964187 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:21.964219 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 12:21:21.964345 22659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38271: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38271: connect: Connection refused (error 111)
I20260925 12:21:21.964816 22741 hybrid_clock.cc:648] HybridClock initialized: now 1790338881964801 us; error 28 us; skew 500 ppm
Sep 25 12:21:21 dist-test-slave-clmx krb5kdc[22578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:21.966841 22741 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 12:21:21.967293 22741 webserver.cc:460] Webserver started at http://127.0.98.60:41183/ using document root <none> and password file <none>
I20260925 12:21:21.967510 22741 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:21.967581 22741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:21.967733 22741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:21.968645 22741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-2/data/instance:
uuid: "f6a565d9b67a43e3a89da12d4536f9ce"
format_stamp: "Formatted at 2026-09-25 12:21:21 on dist-test-slave-clmx"
server_key: "0c8074554676f0a1c54d290be353f46b"
server_key_iv: "4aa8c48842ce7ee7711a19f9d328f071"
server_key_version: "encryptionkey@0"
I20260925 12:21:21.968976 22741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-2/wal/instance:
uuid: "f6a565d9b67a43e3a89da12d4536f9ce"
format_stamp: "Formatted at 2026-09-25 12:21:21 on dist-test-slave-clmx"
server_key: "0c8074554676f0a1c54d290be353f46b"
server_key_iv: "4aa8c48842ce7ee7711a19f9d328f071"
server_key_version: "encryptionkey@0"
I20260925 12:21:21.970080 22741 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:21:21.970708 22756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:21.970851 22741 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:21:21.970933 22741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-2/wal
uuid: "f6a565d9b67a43e3a89da12d4536f9ce"
format_stamp: "Formatted at 2026-09-25 12:21:21 on dist-test-slave-clmx"
server_key: "0c8074554676f0a1c54d290be353f46b"
server_key_iv: "4aa8c48842ce7ee7711a19f9d328f071"
server_key_version: "encryptionkey@0"
I20260925 12:21:21.971009 22741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:21.985562 22741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:21.987143 22741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:21.987277 22741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:21.990839 22741 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38271
I20260925 12:21:21.990865 22808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38271 every 8 connection(s)
I20260925 12:21:21.991127 22741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-2/data/info.pb
I20260925 12:21:21.991926 22809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:21.992332 22734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } attempt: 2
I20260925 12:21:21.993934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 22741
I20260925 12:21:21.994014 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338625091267-392-0/minicluster-data/master-2/wal/instance
I20260925 12:21:21.993922 22809 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:21:21.994211 392 external_mini_cluster.cc:1428] Setting key 26aa5e7f6c5cda8bef670321c979de41
Sep 25 12:21:21 dist-test-slave-clmx krb5kdc[22578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 12:21:21 dist-test-slave-clmx krb5kdc[22578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 12:21:21 dist-test-slave-clmx krb5kdc[22578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 12:21:22.001188 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.992576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38271 (local address 127.0.0.1:53876)
0925 12:21:21.992622 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:21.992634 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:21:21.992680 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:21:21.993429 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:21:21.993430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:21:21.993601 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:21:21.993664 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:21.993667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.995119 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.995120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:21.995829 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:21.995832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.996512 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.996513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:21.996568 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:21.996911 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:21:21.996916 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:21:21.998132 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:21:22.000114 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:22.000115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:22.000116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:22.000269 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:22.000506 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:22.000507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:22.000507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:22.000566 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:22.000975 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:21:22.000976 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:21:22.001016 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 12:21:22.001060 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 12:21:22.001134 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 12:21:22.001820 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.992716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53876 (local address 127.0.98.60:38271)
0925 12:21:21.992864 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:21.992871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:21.992892 (+ 21us) server_negotiation.cc:407] Connection header received
0925 12:21:21.993262 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:21.993268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:21.993307 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:21.993410 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:21.993752 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.995007 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.995900 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.996423 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.996535 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:21.998227 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:21.998233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:21.998234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:21.998246 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:22.000007 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.000354 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.000356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.000357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.000434 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.000624 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.000624 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.000625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.000893 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:22.001322 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:22.001384 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 12:21:22.001492 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
I20260925 12:21:22.002161 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.994531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55380 (local address 127.0.98.62:46121)
0925 12:21:21.994573 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:21.994574 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:21.994733 (+ 159us) server_negotiation.cc:407] Connection header received
0925 12:21:21.994822 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:21.994823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:21.994832 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:21.994877 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:21.995383 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.996009 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.996756 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.997001 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.997084 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:21.999037 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:21.999038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:21.999038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:21.999045 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:22.000548 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.000978 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.000979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.000979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.001029 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.001241 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.001242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.001242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.001363 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:22.001787 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:22.001984 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 12:21:22.002065 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 12:21:22.002707 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.994491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46121 (local address 127.0.0.1:55380)
0925 12:21:21.994636 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:21.994665 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 12:21:21.994744 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:21:21.994896 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:21:21.994898 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:21:21.995212 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:21:21.995296 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:21.995297 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.996090 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.996091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:21.996687 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:21.996689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.997120 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.997122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:21.997188 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:21.997545 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:21:21.997552 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:21:21.998941 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:21:22.000618 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:22.000625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:22.000627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:22.000853 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:22.001106 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:22.001107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:22.001107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:22.001166 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:22.001779 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:21:22.001785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:21:22.001834 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 12:21:22.002466 (+ 632us) client_negotiation.cc:240] Negotiation successful
0925 12:21:22.002559 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260925 12:21:22.003137 22809 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:21:22.003830 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:21.994638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53878 (local address 127.0.98.60:38271)
0925 12:21:21.994783 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:21.994785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:21.994854 (+ 69us) server_negotiation.cc:407] Connection header received
0925 12:21:21.994938 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:21.994939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:21.994951 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:21.995009 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:21.995742 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.996387 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.997367 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:21.997636 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:21.997693 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:22.000518 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:22.000519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:22.000519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:22.000526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:22.002211 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.002608 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.002609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.002609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.002673 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.003120 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.003121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.003121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.003238 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:22.003382 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:22.003593 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 12:21:22.003692 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260925 12:21:22.004009 22734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd: Bootstrap starting.
I20260925 12:21:22.004616 22734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:22.004895 22734 log.cc:824] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:22.005558 22734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd: No bootstrap required, opened a new log
Sep 25 12:21:22 dist-test-slave-clmx krb5kdc[22578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 12:21:22.007251 22734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff32f4a568c747268682b1091c8fb16b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } } peers { permanent_uuid: "6068874d333e4b1da875b017fd80accd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } } peers { permanent_uuid: "f6a565d9b67a43e3a89da12d4536f9ce" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } }
I20260925 12:21:22.007393 22734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6068874d333e4b1da875b017fd80accd, State: Initialized, Role: FOLLOWER
I20260925 12:21:22.007471 22734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff32f4a568c747268682b1091c8fb16b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } } peers { permanent_uuid: "6068874d333e4b1da875b017fd80accd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } } peers { permanent_uuid: "f6a565d9b67a43e3a89da12d4536f9ce" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } }
I20260925 12:21:22.007822 22734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd [sys.catalog]: configured and running, proceeding with master startup.
W20260925 12:21:22.009510 22827 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:21:22.009618 22827 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:21:22.009570 22816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff32f4a568c747268682b1091c8fb16b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } } peers { permanent_uuid: "6068874d333e4b1da875b017fd80accd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } } peers { permanent_uuid: "f6a565d9b67a43e3a89da12d4536f9ce" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } } }
Sep 25 12:21:22 dist-test-slave-clmx krb5kdc[22578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:21:22.009752 22816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:21:22.010329 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:22.003355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42703 (local address 127.0.0.1:41124)
0925 12:21:22.003472 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:22.003482 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:21:22.003542 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:21:22.003737 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:21:22.003738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:21:22.003878 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:21:22.003928 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:22.003929 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.004833 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.004834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:22.005484 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:22.005486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.005909 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.005910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:22.005975 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:22.006360 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:21:22.006365 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:21:22.007561 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:21:22.009404 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:22.009405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:22.009405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:22.009603 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:22.009787 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:22.009788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:22.009788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:22.009840 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:22.010068 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:21:22.010068 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:21:22.010111 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 12:21:22.010171 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 12:21:22.010252 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260925 12:21:22.010619 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:22.003412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.0.98.61:42703)
0925 12:21:22.003447 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.003449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.003520 (+ 71us) server_negotiation.cc:407] Connection header received
0925 12:21:22.003621 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.003622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.003649 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.003677 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.004143 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.004635 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.005569 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.005837 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.005899 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:22.007687 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:22.007688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:22.007688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:22.007694 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:22.009319 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.009675 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.009676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.009677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.009722 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.009886 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.009887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.009887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.010001 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:22.010073 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:22.010485 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 12:21:22.010547 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:21:22.010923 22809 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:21:22.010175 22666 master_runner.cc:186] Error getting master registration for 127.0.98.62:46121: OK, instance_id { permanent_uuid: "ff32f4a568c747268682b1091c8fb16b" instance_seqno: 1790338881698968 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 12:21:22.012966 22666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 12:21:22.014125 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:22.004990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55388 (local address 127.0.98.62:46121)
0925 12:21:22.005020 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.005021 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.005630 (+ 609us) server_negotiation.cc:407] Connection header received
0925 12:21:22.005704 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.005704 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.005712 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.005736 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.006297 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.006697 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.007626 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.007845 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.007893 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:22.010404 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:22.010405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:22.010405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:22.010412 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:22.012937 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.013309 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.013310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.013311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.013369 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.013606 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.013608 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.013608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.013727 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:22.013865 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:22.013956 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 12:21:22.014025 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 25 12:21:22 dist-test-slave-clmx krb5kdc[22578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 12:21:22.018142 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:22.011083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38271 (local address 127.0.0.1:53884)
0925 12:21:22.011158 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:22.011166 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:21:22.011199 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:21:22.011350 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:21:22.011351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:21:22.011505 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:21:22.011544 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:22.011545 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.012379 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.012379 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:22.013077 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:22.013079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.013512 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.013512 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:22.013559 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:22.014157 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:21:22.014162 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:21:22.015322 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:21:22.017012 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:22.017013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:22.017013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:22.017197 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:22.017446 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:22.017447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:22.017447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:22.017507 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:22.017787 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:21:22.017788 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:21:22.017832 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 12:21:22.017907 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 12:21:22.017998 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 12:21:22.018229 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:22.011162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53884 (local address 127.0.98.60:38271)
0925 12:21:22.011197 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.011199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.011208 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:21:22.011255 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.011256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.011264 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.011303 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.011610 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.012294 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.013152 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.013459 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.013514 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:22.015440 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:22.015441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:22.015441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:22.015447 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:22.016939 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.017297 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.017298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.017298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.017376 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.017567 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.017569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.017569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.017722 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:22.017793 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:22.017943 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 12:21:22.018065 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":19}
Sep 25 12:21:22 dist-test-slave-clmx krb5kdc[22578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 12:21:22.019377 22809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce: Bootstrap starting.
I20260925 12:21:22.020038 22809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:22.020370 22809 log.cc:824] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:22.021237 22809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce: No bootstrap required, opened a new log
I20260925 12:21:22.022614 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:22.014723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41138 (local address 127.0.98.61:42703)
0925 12:21:22.014751 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.014753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.014759 (+ 6us) server_negotiation.cc:407] Connection header received
0925 12:21:22.014776 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.014777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.014784 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.014844 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.015399 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.016004 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.016920 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.017205 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.017271 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:22.019857 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:22.019858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:22.019859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:22.019865 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:22.021416 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.021791 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.021792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.021793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.021865 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.022075 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.022076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.022076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.022190 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:22.022351 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:22.022448 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 12:21:22.022523 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 12:21:22.023399 22809 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff32f4a568c747268682b1091c8fb16b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } } peers { permanent_uuid: "6068874d333e4b1da875b017fd80accd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } } peers { permanent_uuid: "f6a565d9b67a43e3a89da12d4536f9ce" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } }
I20260925 12:21:22.023607 22809 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a565d9b67a43e3a89da12d4536f9ce, State: Initialized, Role: FOLLOWER
I20260925 12:21:22.023734 22809 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff32f4a568c747268682b1091c8fb16b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } } peers { permanent_uuid: "6068874d333e4b1da875b017fd80accd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } } peers { permanent_uuid: "f6a565d9b67a43e3a89da12d4536f9ce" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } }
I20260925 12:21:22.023933 22830 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff32f4a568c747268682b1091c8fb16b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } } peers { permanent_uuid: "6068874d333e4b1da875b017fd80accd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } } peers { permanent_uuid: "f6a565d9b67a43e3a89da12d4536f9ce" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } } }
I20260925 12:21:22.023985 22809 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:22.024056 22830 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:21:22.024219 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 12:21:22.025893 22841 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:21:22.025954 22841 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:21:22.025152 22633 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ff32f4a568c747268682b1091c8fb16b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55388
I20260925 12:21:22.026418 22741 master_runner.cc:186] Error getting master registration for 127.0.98.62:46121: OK, instance_id { permanent_uuid: "ff32f4a568c747268682b1091c8fb16b" instance_seqno: 1790338881698968 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 12:21:22.027832 22741 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 12:21:22.039480 22659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } attempt: 2
Sep 25 12:21:22 dist-test-slave-clmx krb5kdc[22578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 12:21:22.046485 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:22.039702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38271 (local address 127.0.0.1:53898)
0925 12:21:22.039740 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:22.039748 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:21:22.039777 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:21:22.039907 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:21:22.039908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:21:22.040062 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:21:22.040096 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:22.040098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.040881 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.040882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:22.041573 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:22.041574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.041984 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.041985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:22.042051 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:22.042438 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:21:22.042443 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:21:22.043790 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:21:22.045557 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:22.045557 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:22.045558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:22.045751 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:22.045925 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:22.045926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:22.045926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:22.045989 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:22.046235 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:21:22.046236 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:21:22.046273 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 12:21:22.046332 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 12:21:22.046405 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 12:21:22.046809 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:22.039753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53898 (local address 127.0.98.60:38271)
0925 12:21:22.039792 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.039794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.039800 (+ 6us) server_negotiation.cc:407] Connection header received
0925 12:21:22.039818 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.039818 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.039837 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.039890 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.040153 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.040805 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.041658 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.041920 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.041978 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:22.043877 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:22.043878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:22.043878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:22.043884 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:22.045449 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.045820 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.045820 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.045821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.045874 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.046043 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.046044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.046044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.046173 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:22.046229 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:22.046668 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 12:21:22.046735 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 12:21:22.048506 22659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b: Bootstrap starting.
I20260925 12:21:22.049166 22659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:22.049410 22659 log.cc:824] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:22.050122 22659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b: No bootstrap required, opened a new log
I20260925 12:21:22.051395 22659 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff32f4a568c747268682b1091c8fb16b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } } peers { permanent_uuid: "6068874d333e4b1da875b017fd80accd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } } peers { permanent_uuid: "f6a565d9b67a43e3a89da12d4536f9ce" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } }
I20260925 12:21:22.051498 22659 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff32f4a568c747268682b1091c8fb16b, State: Initialized, Role: FOLLOWER
I20260925 12:21:22.051568 22659 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff32f4a568c747268682b1091c8fb16b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } } peers { permanent_uuid: "6068874d333e4b1da875b017fd80accd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } } peers { permanent_uuid: "f6a565d9b67a43e3a89da12d4536f9ce" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } }
I20260925 12:21:22.051893 22659 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:22.051946 22633 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 12:21:22.052006 22633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:22.052757 22633 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff32f4a568c747268682b1091c8fb16b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } } peers { permanent_uuid: "6068874d333e4b1da875b017fd80accd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } } peers { permanent_uuid: "f6a565d9b67a43e3a89da12d4536f9ce" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } }
I20260925 12:21:22.052958 22633 leader_election.cc:288] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [CANDIDATE]: Term 1 election: Requested vote from peers 6068874d333e4b1da875b017fd80accd (127.0.98.61:42703), f6a565d9b67a43e3a89da12d4536f9ce (127.0.98.60:38271)
I20260925 12:21:22.053316 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff32f4a568c747268682b1091c8fb16b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6068874d333e4b1da875b017fd80accd"
I20260925 12:21:22.053360 22783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff32f4a568c747268682b1091c8fb16b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f6a565d9b67a43e3a89da12d4536f9ce"
I20260925 12:21:22.053452 22708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:22.053454 22783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:22.054204 22783 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff32f4a568c747268682b1091c8fb16b in term 1.
I20260925 12:21:22.054243 22708 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff32f4a568c747268682b1091c8fb16b in term 1.
I20260925 12:21:22.054446 22607 leader_election.cc:302] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6068874d333e4b1da875b017fd80accd, ff32f4a568c747268682b1091c8fb16b; no voters:
I20260925 12:21:22.051901 22844 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff32f4a568c747268682b1091c8fb16b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } } peers { permanent_uuid: "6068874d333e4b1da875b017fd80accd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } } peers { permanent_uuid: "f6a565d9b67a43e3a89da12d4536f9ce" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } } }
I20260925 12:21:22.055410 22844 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:21:22.055487 22844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:21:22.055559 22844 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [term 1 LEADER]: Becoming Leader. State: Replica: ff32f4a568c747268682b1091c8fb16b, State: Running, Role: LEADER
I20260925 12:21:22.055716 22844 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff32f4a568c747268682b1091c8fb16b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } } peers { permanent_uuid: "6068874d333e4b1da875b017fd80accd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } } peers { permanent_uuid: "f6a565d9b67a43e3a89da12d4536f9ce" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } }
I20260925 12:21:22.056126 22844 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff32f4a568c747268682b1091c8fb16b. Latest consensus state: current_term: 1 leader_uuid: "ff32f4a568c747268682b1091c8fb16b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff32f4a568c747268682b1091c8fb16b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } } peers { permanent_uuid: "6068874d333e4b1da875b017fd80accd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } } peers { permanent_uuid: "f6a565d9b67a43e3a89da12d4536f9ce" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } } }
I20260925 12:21:22.056236 22844 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:22.056855 22591 master_runner.cc:263] No leader master found from master ff32f4a568c747268682b1091c8fb16b
I20260925 12:21:22.056905 22591 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 12:21:22.057422 22855 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:21:22.057469 22855 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:21:22.060981 22862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:21:22.061125 22862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:21:22.068473 22783 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce [term 1 FOLLOWER]: Refusing update from remote peer ff32f4a568c747268682b1091c8fb16b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:21:22.068763 22856 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a565d9b67a43e3a89da12d4536f9ce" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:21:22.069900 22830 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff32f4a568c747268682b1091c8fb16b. Latest consensus state: current_term: 1 leader_uuid: "ff32f4a568c747268682b1091c8fb16b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff32f4a568c747268682b1091c8fb16b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } } peers { permanent_uuid: "6068874d333e4b1da875b017fd80accd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } } peers { permanent_uuid: "f6a565d9b67a43e3a89da12d4536f9ce" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } } }
I20260925 12:21:22.069990 22830 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:21:22.070277 22708 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd [term 1 FOLLOWER]: Refusing update from remote peer ff32f4a568c747268682b1091c8fb16b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:21:22.070492 22856 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6068874d333e4b1da875b017fd80accd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:21:22.071594 22816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff32f4a568c747268682b1091c8fb16b. Latest consensus state: current_term: 1 leader_uuid: "ff32f4a568c747268682b1091c8fb16b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff32f4a568c747268682b1091c8fb16b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } } peers { permanent_uuid: "6068874d333e4b1da875b017fd80accd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } } peers { permanent_uuid: "f6a565d9b67a43e3a89da12d4536f9ce" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } } }
I20260925 12:21:22.071688 22816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:21:22.075968 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:22.057939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55398 (local address 127.0.98.62:46121)
0925 12:21:22.057966 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.057968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.059781 (+ 1813us) server_negotiation.cc:407] Connection header received
0925 12:21:22.059868 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.059868 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.059877 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.059902 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.061737 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.062393 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.065218 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.065512 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.065561 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:22.072419 (+ 6858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:22.072421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:22.072421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:22.072427 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:22.073997 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.075187 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.075188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.075189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.075240 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.075415 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.075416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.075416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.075520 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:22.075637 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:22.075820 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 12:21:22.075889 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 12:21:22.077384 22856 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff32f4a568c747268682b1091c8fb16b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff32f4a568c747268682b1091c8fb16b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } } peers { permanent_uuid: "6068874d333e4b1da875b017fd80accd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } } peers { permanent_uuid: "f6a565d9b67a43e3a89da12d4536f9ce" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } } }
I20260925 12:21:22.077468 22856 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:22.077838 22830 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff32f4a568c747268682b1091c8fb16b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff32f4a568c747268682b1091c8fb16b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } } peers { permanent_uuid: "6068874d333e4b1da875b017fd80accd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } } peers { permanent_uuid: "f6a565d9b67a43e3a89da12d4536f9ce" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } } }
I20260925 12:21:22.077924 22830 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:21:22.078166 22867 mvcc.cc:205] Tried to move back new op lower bound from 7333228060949983232 to 7333228060901089280. Current Snapshot: MvccSnapshot[applied={T|T < 7333228060949983232}]
I20260925 12:21:22.078261 22868 mvcc.cc:205] Tried to move back new op lower bound from 7333228060949983232 to 7333228060901089280. Current Snapshot: MvccSnapshot[applied={T|T < 7333228060949983232}]
I20260925 12:21:22.081352 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:22.057785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.0.98.61:42703)
0925 12:21:22.057817 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.057819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.063491 (+ 5672us) server_negotiation.cc:407] Connection header received
0925 12:21:22.063606 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.063607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.063642 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.063669 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.066003 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.066667 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.070157 (+ 3490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.070392 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.070433 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:22.076364 (+ 5931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:22.076365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:22.076365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:22.076372 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:22.077868 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.080051 (+ 2183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.080052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.080053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.080102 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.080278 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.080278 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.080279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.080384 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:22.080452 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:22.081202 (+ 750us) server_negotiation.cc:299] Negotiation successful
0925 12:21:22.081274 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 12:21:22.081964 22844 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff32f4a568c747268682b1091c8fb16b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff32f4a568c747268682b1091c8fb16b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } } peers { permanent_uuid: "6068874d333e4b1da875b017fd80accd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } } peers { permanent_uuid: "f6a565d9b67a43e3a89da12d4536f9ce" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } } }
I20260925 12:21:22.082032 22844 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:22.082080 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:22.058284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.0.98.60:38271)
0925 12:21:22.058313 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.058315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.064408 (+ 6093us) server_negotiation.cc:407] Connection header received
0925 12:21:22.064496 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.064497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.064506 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.064533 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.068655 (+ 4122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.069266 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.070847 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:22.071085 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:22.071125 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:22.078240 (+ 7115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:22.078241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:22.078241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:22.078247 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:22.079715 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.080634 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.080635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.080635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.080685 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:22.080855 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:22.080855 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:22.080856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:22.080963 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:22.081031 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:22.081934 (+ 903us) server_negotiation.cc:299] Negotiation successful
0925 12:21:22.082002 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 12:21:22.082540 22816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff32f4a568c747268682b1091c8fb16b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff32f4a568c747268682b1091c8fb16b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46121 } } peers { permanent_uuid: "6068874d333e4b1da875b017fd80accd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42703 } } peers { permanent_uuid: "f6a565d9b67a43e3a89da12d4536f9ce" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38271 } } }
I20260925 12:21:22.082623 22816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:21:22.083065 22870 mvcc.cc:205] Tried to move back new op lower bound from 7333228060949983232 to 7333228060901089280. Current Snapshot: MvccSnapshot[applied={T|T < 7333228060949983232}]
I20260925 12:21:22.084007 22862 catalog_manager.cc:1347] Generated new cluster ID: 2cf0ada4e6e046b5bf46a0010f28925c
I20260925 12:21:22.084085 22862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:21:22.091136 22862 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:21:22.091853 22862 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:21:22.111696 22862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff32f4a568c747268682b1091c8fb16b: Generated new TSK 0
I20260925 12:21:22.111997 22862 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 12:21:22.133899 22663 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:22.132273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55404 (local address 127.0.98.62:46121)
0925 12:21:22.132315 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.132317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.133231 (+ 914us) server_negotiation.cc:407] Connection header received
0925 12:21:22.133339 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.133340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.133351 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.133383 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.133834 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55404: BlockingRecv error: recv got EOF from 127.0.0.1:55404 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 12:21:22.134780 22738 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:22.132696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41150 (local address 127.0.98.61:42703)
0925 12:21:22.132733 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.132735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.134283 (+ 1548us) server_negotiation.cc:407] Connection header received
0925 12:21:22.134374 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.134375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.134384 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.134412 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.134724 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41150: BlockingRecv error: recv got EOF from 127.0.0.1:41150 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 12:21:22.135586 22814 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:22.132979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53912 (local address 127.0.98.60:38271)
0925 12:21:22.133016 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.133018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.135103 (+ 2085us) server_negotiation.cc:407] Connection header received
0925 12:21:22.135194 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.135195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.135205 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.135234 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.135520 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53912: BlockingRecv error: recv got EOF from 127.0.0.1:53912 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 12:21:22.140336 22738 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:22.139576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.0.98.61:42703)
0925 12:21:22.139607 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.139608 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.139846 (+ 238us) server_negotiation.cc:407] Connection header received
0925 12:21:22.139943 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.139944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.139960 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.139989 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.140296 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41160: BlockingRecv error: recv got EOF from 127.0.0.1:41160 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 12:21:22.143337 22738 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:22.142611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41164 (local address 127.0.98.61:42703)
0925 12:21:22.142642 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.142644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.142862 (+ 218us) server_negotiation.cc:407] Connection header received
0925 12:21:22.142957 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.142959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.142968 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.142996 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.143294 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41164: BlockingRecv error: recv got EOF from 127.0.0.1:41164 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 12:21:22.147238 22738 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:22.146530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41170 (local address 127.0.98.61:42703)
0925 12:21:22.146571 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.146573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.146774 (+ 201us) server_negotiation.cc:407] Connection header received
0925 12:21:22.146866 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.146867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.146875 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.146902 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.147201 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41170: BlockingRecv error: recv got EOF from 127.0.0.1:41170 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 12:21:22.153221 22738 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:22.152438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41184 (local address 127.0.98.61:42703)
0925 12:21:22.152467 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.152469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.152755 (+ 286us) server_negotiation.cc:407] Connection header received
0925 12:21:22.152865 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.152866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.152875 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.152902 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.153185 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41184: BlockingRecv error: recv got EOF from 127.0.0.1:41184 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 12:21:22.163074 22738 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:22.162320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41188 (local address 127.0.98.61:42703)
0925 12:21:22.162350 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.162352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.162597 (+ 245us) server_negotiation.cc:407] Connection header received
0925 12:21:22.162694 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.162695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.162704 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.162732 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.163034 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41188: BlockingRecv error: recv got EOF from 127.0.0.1:41188 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 12:21:22.181094 22738 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:22.180362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41204 (local address 127.0.98.61:42703)
0925 12:21:22.180397 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.180399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.180620 (+ 221us) server_negotiation.cc:407] Connection header received
0925 12:21:22.180703 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.180704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.180712 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.180738 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.181046 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41204: BlockingRecv error: recv got EOF from 127.0.0.1:41204 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 12:21:22.215209 22738 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:22.214418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.0.98.61:42703)
0925 12:21:22.214481 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.214483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.214603 (+ 120us) server_negotiation.cc:407] Connection header received
0925 12:21:22.214682 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.214683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.214692 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.214743 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.215138 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41218: BlockingRecv error: recv got EOF from 127.0.0.1:41218 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260925 12:21:22.281019 22738 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:22.280398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.0.98.61:42703)
0925 12:21:22.280462 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.280465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.280565 (+ 100us) server_negotiation.cc:407] Connection header received
0925 12:21:22.280614 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.280615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.280624 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.280662 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.280952 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41234: BlockingRecv error: recv got EOF from 127.0.0.1:41234 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 12:21:22.410616 22738 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:22.409978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41246 (local address 127.0.98.61:42703)
0925 12:21:22.410032 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.410033 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.410151 (+ 118us) server_negotiation.cc:407] Connection header received
0925 12:21:22.410196 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.410197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.410206 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.410266 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.410546 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41246: BlockingRecv error: recv got EOF from 127.0.0.1:41246 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260925 12:21:22.669418 22738 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:22.668469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55204 (local address 127.0.98.61:42703)
0925 12:21:22.668817 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:22.668819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:22.668829 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:21:22.668872 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:22.668873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:22.668883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:22.668934 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:22.669353 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55204: BlockingRecv error: recv got EOF from 127.0.0.1:55204 (error 108)
Metrics: {"server-negotiator.queue_time_us":323}
I20260925 12:21:23.010043 22827 catalog_manager.cc:1259] Loaded cluster ID: 2cf0ada4e6e046b5bf46a0010f28925c
I20260925 12:21:23.010103 22827 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd: loading cluster ID for follower catalog manager: success
I20260925 12:21:23.010824 22827 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd: acquiring CA information for follower catalog manager: success
I20260925 12:21:23.011061 22827 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6068874d333e4b1da875b017fd80accd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 12:21:23.026335 22841 catalog_manager.cc:1259] Loaded cluster ID: 2cf0ada4e6e046b5bf46a0010f28925c
I20260925 12:21:23.026409 22841 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce: loading cluster ID for follower catalog manager: success
I20260925 12:21:23.027115 22841 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce: acquiring CA information for follower catalog manager: success
I20260925 12:21:23.027386 22841 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f6a565d9b67a43e3a89da12d4536f9ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 12:21:23.185140 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:23.182746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55214 (local address 127.0.98.61:42703)
0925 12:21:23.182856 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:23.182858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:23.182935 (+ 77us) server_negotiation.cc:407] Connection header received
0925 12:21:23.182992 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:23.182993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:23.183040 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:23.183110 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:21:23.183238 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:23.183756 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:23.184434 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:23.184613 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:23.184660 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:23.184806 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:21:23.184913 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:21:23.184969 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:23.185002 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 12:21:23.185018 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":70,"threads_started":1}
I20260925 12:21:23.189239 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:23.186943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50840 (local address 127.0.98.60:38271)
0925 12:21:23.187085 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:23.187087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:23.187093 (+ 6us) server_negotiation.cc:407] Connection header received
0925 12:21:23.187127 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:23.187127 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:23.187158 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:23.187219 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:21:23.187290 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:23.187834 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:23.188509 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:23.188743 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:23.188792 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:23.188929 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:21:23.189011 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:21:23.189063 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:23.189105 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 12:21:23.189121 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260925 12:21:23.193593 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:23.190986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.0.98.60:38271)
0925 12:21:23.191015 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:23.191017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:23.191023 (+ 6us) server_negotiation.cc:407] Connection header received
0925 12:21:23.191103 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:23.191104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:23.191114 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:23.191156 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:21:23.191311 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:23.191998 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:23.192771 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:23.192995 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:23.193054 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:23.193199 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:21:23.193227 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:21:23.193283 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:23.193497 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 12:21:23.193517 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 12:21:23.193907 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:23.190826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55228 (local address 127.0.98.61:42703)
0925 12:21:23.190865 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:23.190867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:23.190956 (+ 89us) server_negotiation.cc:407] Connection header received
0925 12:21:23.191062 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:23.191063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:23.191072 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:23.191117 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:21:23.191258 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:23.191961 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:23.192946 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:23.193229 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:23.193289 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:23.193432 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:21:23.193459 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:21:23.193509 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:23.193802 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 12:21:23.193824 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:21:23.193948 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:23.190788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44712 (local address 127.0.98.62:46121)
0925 12:21:23.190890 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:23.190892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:23.190901 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:21:23.190977 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:23.190978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:23.191027 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:23.191117 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:21:23.191229 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:23.191790 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:23.192827 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:23.193139 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:23.193198 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:23.193387 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:21:23.193493 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:21:23.193549 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:23.193833 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 12:21:23.193857 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":50,"threads_started":1}
I20260925 12:21:23.195353 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 22591
I20260925 12:21:23.200255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 22666
I20260925 12:21:23.205261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 22741
2026-09-25T12:21:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2694 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:21:23 dist-test-slave-clmx krb5kdc[22958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:21:23 dist-test-slave-clmx krb5kdc[22958](info): set up 2 sockets
Sep 25 12:21:23 dist-test-slave-clmx krb5kdc[22958](info): commencing operation
krb5kdc: starting...
W20260925 12:21:24.183494 392 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/test-user.keytab.
Sep 25 12:21:24 dist-test-slave-clmx krb5kdc[22958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:21:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:21:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:21:24.288322 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37011
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37011,127.0.98.61:45129,127.0.98.60:46345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:21:24.361795 22971 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 12:21:24.361963 22971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:24.361979 22971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:24.361995 22971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:24.363018 22971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:24.363050 22971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:24.363063 22971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:24.363082 22971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:24.363094 22971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:24.364485 22971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37011,127.0.98.61:45129,127.0.98.60:46345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:24.364651 22971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:24.364794 22971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:24.369062 22979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:24.369107 22977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:24.369092 22976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:24.369136 22971 server_base.cc:1034] running on GCE node
I20260925 12:21:24.369434 22971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:24.369656 22971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:24.370797 22971 hybrid_clock.cc:648] HybridClock initialized: now 1790338884370786 us; error 24 us; skew 500 ppm
Sep 25 12:21:24 dist-test-slave-clmx krb5kdc[22958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:24.373083 22971 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:21:24.373503 22971 webserver.cc:460] Webserver started at http://127.0.98.62:39077/ using document root <none> and password file <none>
I20260925 12:21:24.373714 22971 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:24.373787 22971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:24.373929 22971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:24.374842 22971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-0/data/instance:
uuid: "3914136d4eaa46ddbf143969a8566501"
format_stamp: "Formatted at 2026-09-25 12:21:24 on dist-test-slave-clmx"
server_key: "8189c7f6f46ff14352ada94b069d0458"
server_key_iv: "ba15f4a1c744dadac440b915d4c1277c"
server_key_version: "encryptionkey@0"
I20260925 12:21:24.375169 22971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-0/wal/instance:
uuid: "3914136d4eaa46ddbf143969a8566501"
format_stamp: "Formatted at 2026-09-25 12:21:24 on dist-test-slave-clmx"
server_key: "8189c7f6f46ff14352ada94b069d0458"
server_key_iv: "ba15f4a1c744dadac440b915d4c1277c"
server_key_version: "encryptionkey@0"
I20260925 12:21:24.376426 22971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:21:24.377068 22986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:24.377254 22971 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:21:24.377337 22971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-0/wal
uuid: "3914136d4eaa46ddbf143969a8566501"
format_stamp: "Formatted at 2026-09-25 12:21:24 on dist-test-slave-clmx"
server_key: "8189c7f6f46ff14352ada94b069d0458"
server_key_iv: "ba15f4a1c744dadac440b915d4c1277c"
server_key_version: "encryptionkey@0"
I20260925 12:21:24.377408 22971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:24.406378 22971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:24.408643 22971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:24.408795 22971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:24.412533 22971 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37011
I20260925 12:21:24.412563 23038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37011 every 8 connection(s)
I20260925 12:21:24.412907 22971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:21:24.413790 23039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:24.415208 23039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:21:24 dist-test-slave-clmx krb5kdc[22958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:21:24.422473 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 22971
I20260925 12:21:24.422561 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-0/wal/instance
I20260925 12:21:24.422755 392 external_mini_cluster.cc:1428] Setting key aba3eddcde45db69788783612cb72e72
I20260925 12:21:24.424240 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.415834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37011 (local address 127.0.0.1:47382)
0925 12:21:24.415983 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.416019 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:21:24.416462 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:21:24.416728 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:21:24.416730 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:21:24.416979 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:21:24.417135 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:24.417139 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.418128 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.418129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:24.418789 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:24.418790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.419267 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.419268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:24.419325 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.419818 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:21:24.419826 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:21:24.421067 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:21:24.422583 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:24.422593 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:24.422595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:24.422856 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:24.423115 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:24.423116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:24.423116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:24.423192 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:24.423654 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:21:24.423657 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:21:24.423717 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 12:21:24.423809 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 12:21:24.423962 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260925 12:21:24.424525 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.415857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47382 (local address 127.0.98.62:37011)
0925 12:21:24.415994 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.416002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.416116 (+ 114us) server_negotiation.cc:407] Connection header received
0925 12:21:24.416600 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.416603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.416621 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.416712 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.417238 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.418016 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.418877 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.419165 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.419280 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.421164 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:24.421170 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:24.421170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:24.421176 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:24.422468 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.422943 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.422946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.422947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.423048 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.423257 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.423258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.423259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.423535 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:24.423613 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:24.424372 (+ 759us) server_negotiation.cc:299] Negotiation successful
0925 12:21:24.424451 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260925 12:21:24.425099 23039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:21:24.425410 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.425304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45129 (local address 127.0.0.1:44878)
0925 12:21:24.425347 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.425385 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 12:21:24.425558 22988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45129: connect: Connection refused (error 111)
I20260925 12:21:24.425941 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.425864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45129 (local address 127.0.0.1:44894)
0925 12:21:24.425906 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.425927 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 12:21:24.426045 23039 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45129: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45129: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 12:21:24.444410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45129
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37011,127.0.98.61:45129,127.0.98.60:46345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:21:24.475263 23039 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } attempt: 1
I20260925 12:21:24.475684 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.475525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45129 (local address 127.0.0.1:44906)
0925 12:21:24.475576 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.475648 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 12:21:24.476051 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.475944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45129 (local address 127.0.0.1:44920)
0925 12:21:24.475993 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.476026 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 12:21:24.476181 23039 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45129: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45129: connect: Connection refused (error 111)
W20260925 12:21:24.519820 23046 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 12:21:24.519989 23046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:24.520004 23046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:24.520042 23046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:24.521126 23046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:24.521165 23046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:24.521178 23046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:24.521191 23046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:24.521238 23046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:24.522617 23046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37011,127.0.98.61:45129,127.0.98.60:46345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:24.522864 23046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:24.523082 23046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:24.527558 23051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:24.527591 23052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:24.527601 23046 server_base.cc:1034] running on GCE node
W20260925 12:21:24.527570 23054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:24.528038 23046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:24.528302 23046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:24.529433 23046 hybrid_clock.cc:648] HybridClock initialized: now 1790338884529415 us; error 34 us; skew 500 ppm
Sep 25 12:21:24 dist-test-slave-clmx krb5kdc[22958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:24.531576 23046 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 12:21:24.531962 23046 webserver.cc:460] Webserver started at http://127.0.98.61:37045/ using document root <none> and password file <none>
I20260925 12:21:24.532182 23046 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:24.532245 23046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:24.532383 23046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:24.533335 23046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-1/data/instance:
uuid: "8e51d1a3bc6341b7a913e813a397cceb"
format_stamp: "Formatted at 2026-09-25 12:21:24 on dist-test-slave-clmx"
server_key: "0671f74eb14878626111799c3cb7cb39"
server_key_iv: "4e36459f4a8e3648f36eb694d3652bb8"
server_key_version: "encryptionkey@0"
I20260925 12:21:24.533667 23046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-1/wal/instance:
uuid: "8e51d1a3bc6341b7a913e813a397cceb"
format_stamp: "Formatted at 2026-09-25 12:21:24 on dist-test-slave-clmx"
server_key: "0671f74eb14878626111799c3cb7cb39"
server_key_iv: "4e36459f4a8e3648f36eb694d3652bb8"
server_key_version: "encryptionkey@0"
I20260925 12:21:24.534854 23046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:21:24.535492 23061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:24.535668 23046 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:24.535769 23046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-1/wal
uuid: "8e51d1a3bc6341b7a913e813a397cceb"
format_stamp: "Formatted at 2026-09-25 12:21:24 on dist-test-slave-clmx"
server_key: "0671f74eb14878626111799c3cb7cb39"
server_key_iv: "4e36459f4a8e3648f36eb694d3652bb8"
server_key_version: "encryptionkey@0"
I20260925 12:21:24.535841 23046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:24.544348 23039 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } attempt: 2
I20260925 12:21:24.544724 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.544611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45129 (local address 127.0.0.1:44934)
0925 12:21:24.544667 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.544699 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 12:21:24.545069 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.544989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45129 (local address 127.0.0.1:44948)
0925 12:21:24.545031 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.545054 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 12:21:24.545177 23039 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45129: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45129: connect: Connection refused (error 111)
I20260925 12:21:24.564965 23046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:24.566654 23046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:24.566749 23046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:24.571201 23046 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45129
I20260925 12:21:24.571297 23113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45129 every 8 connection(s)
I20260925 12:21:24.571604 23046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-1/data/info.pb
I20260925 12:21:24.572561 23114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:24.573995 23114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:21:24.578737 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 23046
I20260925 12:21:24.578845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-1/wal/instance
Sep 25 12:21:24 dist-test-slave-clmx krb5kdc[22958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:21:24.579017 392 external_mini_cluster.cc:1428] Setting key 2c5bdd649b6252484b3b53b6169de113
I20260925 12:21:24.582358 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.574595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47398 (local address 127.0.98.62:37011)
0925 12:21:24.574659 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.574662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.574844 (+ 182us) server_negotiation.cc:407] Connection header received
0925 12:21:24.575223 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.575224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.575234 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.575282 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.575847 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.576314 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.577225 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.577412 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.577459 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.579593 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:24.579595 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:24.579595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:24.579601 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:24.581093 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.581475 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.581476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.581476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.581550 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.581811 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.581811 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.581811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.581902 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:24.581950 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:24.582243 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 12:21:24.582302 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 12:21:24.582633 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.574580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37011 (local address 127.0.0.1:47398)
0925 12:21:24.574729 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.574758 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 12:21:24.575149 (+ 391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:21:24.575340 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:21:24.575343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:21:24.575538 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:21:24.575778 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:24.575782 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.576381 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.576382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:24.577160 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:24.577161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.577490 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.577491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:24.577598 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.577958 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:21:24.577966 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:21:24.579515 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:21:24.581161 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:24.581166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:24.581167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:24.581376 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:24.581615 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:24.581616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:24.581617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:24.581750 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:24.581962 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:21:24.581967 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:21:24.582122 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 12:21:24.582224 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 12:21:24.582379 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260925 12:21:24.583154 23114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Sep 25 12:21:24 dist-test-slave-clmx krb5kdc[22958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260925 12:21:24.590241 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.583390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45129 (local address 127.0.0.1:44950)
0925 12:21:24.583582 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.583593 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:21:24.583670 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:21:24.583825 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:21:24.583825 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:21:24.583981 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:21:24.584033 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:24.584035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.584813 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.584814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:24.585707 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:24.585710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.586148 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.586149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:24.586233 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.586555 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:21:24.586560 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:21:24.587774 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:21:24.589189 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:24.589189 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:24.589190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:24.589370 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:24.589610 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:24.589611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:24.589612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:24.589681 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:24.589936 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:21:24.589937 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:21:24.589971 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 12:21:24.590052 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 12:21:24.590139 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260925 12:21:24.590322 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.583457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44950 (local address 127.0.98.61:45129)
0925 12:21:24.583598 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.583601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.583675 (+ 74us) server_negotiation.cc:407] Connection header received
0925 12:21:24.583731 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.583733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.583743 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.583821 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.584098 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.584734 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.585793 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.586066 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.586118 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.587845 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:24.587851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:24.587852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:24.587857 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:24.589122 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.589469 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.589473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.589473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.589528 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.589746 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.589747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.589747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.589858 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:24.589942 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:24.590080 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 12:21:24.590189 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260925 12:21:24.590799 23114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:21:24.591100 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.591005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46345 (local address 127.0.0.1:53184)
0925 12:21:24.591049 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.591085 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 12:21:24.591259 23063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46345: connect: Connection refused (error 111)
I20260925 12:21:24.591701 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.591575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46345 (local address 127.0.0.1:53200)
0925 12:21:24.591653 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.591680 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 12:21:24.591826 23114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46345: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46345: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 12:21:24.602240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKTlv5c/build/release/bin/kudu
/tmp/dist-test-taskKTlv5c/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46345
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37011,127.0.98.61:45129,127.0.98.60:46345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:21:24.636329 23039 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } attempt: 3
Sep 25 12:21:24 dist-test-slave-clmx krb5kdc[22958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 12:21:24.641004 23114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } attempt: 1
I20260925 12:21:24.641310 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.641200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46345 (local address 127.0.0.1:53216)
0925 12:21:24.641251 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.641282 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 12:21:24.641959 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.641540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46345 (local address 127.0.0.1:53226)
0925 12:21:24.641909 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.641935 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352}
W20260925 12:21:24.642119 23114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46345: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46345: connect: Connection refused (error 111)
I20260925 12:21:24.642612 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.636629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45129 (local address 127.0.0.1:44962)
0925 12:21:24.636668 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.636684 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 12:21:24.636743 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:21:24.636881 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:21:24.636881 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:21:24.637009 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:21:24.637061 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:24.637062 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.637640 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.637641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:24.638152 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:24.638155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.638504 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.638505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:24.638586 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.639020 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:21:24.639024 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:21:24.640106 (+ 1082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:21:24.641604 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:24.641605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:24.641605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:24.641807 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:24.641993 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:24.641994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:24.641994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:24.642053 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:24.642302 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:21:24.642303 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:21:24.642347 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 12:21:24.642417 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 12:21:24.642519 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 12:21:24.642617 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.636678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.0.98.61:45129)
0925 12:21:24.636731 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.636733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.636742 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:21:24.636803 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.636803 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.636812 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.636877 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.637136 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.637561 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.638232 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.638417 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.638507 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.640187 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:24.640188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:24.640188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:24.640195 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:24.641527 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.641880 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.641881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.641881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.641923 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.642104 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.642105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.642105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.642232 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:24.642297 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:24.642439 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 12:21:24.642519 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 12:21:24.643136 23039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:21:24.643424 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.643336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46345 (local address 127.0.0.1:53234)
0925 12:21:24.643378 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.643405 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 12:21:24.643738 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.643672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46345 (local address 127.0.0.1:53244)
0925 12:21:24.643699 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.643725 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260925 12:21:24.643838 23039 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46345: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46345: connect: Connection refused (error 111)
I20260925 12:21:24.674978 23039 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } attempt: 1
I20260925 12:21:24.675328 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.675228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46345 (local address 127.0.0.1:53258)
0925 12:21:24.675279 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.675309 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 12:21:24.675738 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.675654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46345 (local address 127.0.0.1:53262)
0925 12:21:24.675688 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.675722 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 12:21:24.675822 23039 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46345: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46345: connect: Connection refused (error 111)
W20260925 12:21:24.677850 23121 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 12:21:24.677997 23121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:21:24.678011 23121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:21:24.678053 23121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:21:24.679040 23121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:21:24.679073 23121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:21:24.679086 23121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:21:24.679131 23121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:21:24.679153 23121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:21:24.680465 23121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37011,127.0.98.61:45129,127.0.98.60:46345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:21:24.680622 23121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:21:24.680824 23121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:21:24.685000 23126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:24.685045 23129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:21:24.685178 23127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:21:24.685178 23121 server_base.cc:1034] running on GCE node
I20260925 12:21:24.685454 23121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:21:24.685742 23121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:21:24.686895 23121 hybrid_clock.cc:648] HybridClock initialized: now 1790338884686889 us; error 34 us; skew 500 ppm
Sep 25 12:21:24 dist-test-slave-clmx krb5kdc[22958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:21:24.689059 23121 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 12:21:24.689491 23121 webserver.cc:460] Webserver started at http://127.0.98.60:39431/ using document root <none> and password file <none>
I20260925 12:21:24.689692 23121 fs_manager.cc:359] Metadata directory not provided
I20260925 12:21:24.689755 23121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:21:24.689888 23121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:21:24.690820 23121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-2/data/instance:
uuid: "ed21138026b24b1e9a4d536d7b6c8bb6"
format_stamp: "Formatted at 2026-09-25 12:21:24 on dist-test-slave-clmx"
server_key: "5dfd1003edca394b71e9aeaac3be7e3e"
server_key_iv: "968378ab03c5db3e8ae24dc3c259830a"
server_key_version: "encryptionkey@0"
I20260925 12:21:24.691143 23121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-2/wal/instance:
uuid: "ed21138026b24b1e9a4d536d7b6c8bb6"
format_stamp: "Formatted at 2026-09-25 12:21:24 on dist-test-slave-clmx"
server_key: "5dfd1003edca394b71e9aeaac3be7e3e"
server_key_iv: "968378ab03c5db3e8ae24dc3c259830a"
server_key_version: "encryptionkey@0"
I20260925 12:21:24.692394 23121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:21:24.693066 23136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:21:24.693228 23121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:21:24.693316 23121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-2/wal
uuid: "ed21138026b24b1e9a4d536d7b6c8bb6"
format_stamp: "Formatted at 2026-09-25 12:21:24 on dist-test-slave-clmx"
server_key: "5dfd1003edca394b71e9aeaac3be7e3e"
server_key_iv: "968378ab03c5db3e8ae24dc3c259830a"
server_key_version: "encryptionkey@0"
I20260925 12:21:24.693388 23121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:21:24.710266 23114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } attempt: 2
I20260925 12:21:24.710666 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.710530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46345 (local address 127.0.0.1:53268)
0925 12:21:24.710606 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.710640 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 12:21:24.711032 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.710904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46345 (local address 127.0.0.1:53282)
0925 12:21:24.710958 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.710997 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 12:21:24.711113 23114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46345: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46345: connect: Connection refused (error 111)
I20260925 12:21:24.719453 23121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:21:24.721065 23121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:21:24.721213 23121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:21:24.725255 23121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46345
I20260925 12:21:24.725490 23188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46345 every 8 connection(s)
I20260925 12:21:24.725708 23121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-2/data/info.pb
I20260925 12:21:24.726155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKTlv5c/build/release/bin/kudu as pid 23121
I20260925 12:21:24.726264 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKTlv5c/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338625091267-392-0/minicluster-data/master-2/wal/instance
I20260925 12:21:24.726436 392 external_mini_cluster.cc:1428] Setting key 77d73a29c7e013615bc38480e9945414
I20260925 12:21:24.726678 23189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:21:24.728734 23189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:21:24 dist-test-slave-clmx krb5kdc[22958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 12:21:24 dist-test-slave-clmx krb5kdc[22958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:21:24.736599 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.729376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47414 (local address 127.0.98.62:37011)
0925 12:21:24.729439 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.729441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.729628 (+ 187us) server_negotiation.cc:407] Connection header received
0925 12:21:24.729655 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.729656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.729667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.729727 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.730224 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.730745 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.731602 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.731817 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.731865 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.733975 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:24.733976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:24.733977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:24.733984 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:24.735396 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.735802 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.735803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.735804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.735857 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.736061 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.736062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.736062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.736169 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:24.736233 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:24.736435 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 12:21:24.736508 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 12:21:24.736892 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.729309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37011 (local address 127.0.0.1:47414)
0925 12:21:24.729502 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.729538 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:21:24.729591 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:21:24.729775 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:21:24.729777 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:21:24.730076 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:21:24.730149 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:24.730151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.730821 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.730822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:24.731521 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:24.731524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.731910 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.731912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:24.731986 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.732344 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:21:24.732352 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:21:24.733897 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:21:24.735465 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:24.735471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:24.735473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:24.735742 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:24.735917 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:24.735917 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:24.735918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:24.736002 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:24.736231 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:21:24.736235 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:21:24.736281 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 12:21:24.736400 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 12:21:24.736545 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":94,"threads_started":1}
I20260925 12:21:24.736896 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.726906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.0.98.60:46345)
0925 12:21:24.727098 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.727106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.727133 (+ 27us) server_negotiation.cc:407] Connection header received
0925 12:21:24.727713 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.727721 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.727777 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.727907 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.728456 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.729649 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.730554 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.731058 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.731135 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.733465 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:24.733472 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:24.733473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:24.733479 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:24.735262 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.735693 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.735695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.735695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.735807 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.736027 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.736027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.736027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.736280 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:24.736422 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:24.736528 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 12:21:24.736592 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
I20260925 12:21:24.737579 23189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:21:24 dist-test-slave-clmx krb5kdc[22958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 25 12:21:24 dist-test-slave-clmx krb5kdc[22958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:21:24.743973 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.737880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44974 (local address 127.0.98.61:45129)
0925 12:21:24.737918 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.737920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.737977 (+ 57us) server_negotiation.cc:407] Connection header received
0925 12:21:24.738036 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.738037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.738046 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.738107 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.738368 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.738795 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.739525 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.739802 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.739881 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.741397 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:24.741399 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:24.741399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:24.741404 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:24.742859 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.743212 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.743213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.743214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.743266 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.743467 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.743468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.743469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.743578 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:24.743667 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:24.743818 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 12:21:24.743894 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:21:24.743973 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.737827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45129 (local address 127.0.0.1:44974)
0925 12:21:24.737914 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.737925 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:21:24.737969 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:21:24.738113 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:21:24.738113 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:21:24.738253 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:21:24.738292 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:24.738293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.738882 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.738883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:24.739438 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:24.739440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.739882 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.739883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:24.739949 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.740254 (+ 305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:21:24.740258 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:21:24.741322 (+ 1064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:21:24.742957 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:24.742958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:24.742958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:24.743126 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:24.743340 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:24.743341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:24.743341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:24.743396 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:24.743670 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:21:24.743671 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:21:24.743710 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 12:21:24.743793 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 12:21:24.743890 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 12:21:24.744266 23189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:21:24.744524 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.737809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47420 (local address 127.0.98.62:37011)
0925 12:21:24.737840 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.737841 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.737849 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:21:24.737866 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.737866 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.737874 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.737921 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.738405 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.738830 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.739632 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.739825 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.739895 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.741996 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:24.741997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:24.741998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:24.742003 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:24.743259 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.743587 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.743588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.743588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.743694 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.743899 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.743900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.743901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.743996 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:24.744193 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:24.744391 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 12:21:24.744456 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 25 12:21:24 dist-test-slave-clmx krb5kdc[22958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 12:21:24 dist-test-slave-clmx krb5kdc[22958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 12:21:24.750948 23039 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } attempt: 2
I20260925 12:21:24.751037 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.744476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46345 (local address 127.0.0.1:53302)
0925 12:21:24.744536 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.744545 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:21:24.744590 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:21:24.744713 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:21:24.744714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:21:24.744872 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:21:24.744899 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:24.744901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.745691 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.745691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:24.746140 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:24.746141 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.746493 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.746494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:24.746538 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.746911 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:21:24.746915 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:21:24.748066 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:21:24.750160 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:24.750161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:24.750162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:24.750371 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:24.750550 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:24.750551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:24.750551 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:24.750607 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:24.750792 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:21:24.750793 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:21:24.750833 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 12:21:24.750897 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 12:21:24.750964 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 12:21:24.751443 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.744511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53302 (local address 127.0.98.60:46345)
0925 12:21:24.744539 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.744541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.744570 (+ 29us) server_negotiation.cc:407] Connection header received
0925 12:21:24.744647 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.744647 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.744654 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.744688 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.744968 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.745608 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.746215 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.746418 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.746484 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.748149 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:24.748150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:24.748151 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:24.748157 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:24.750088 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.750442 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.750443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.750443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.750487 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.750664 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.750664 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.750665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.750739 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:24.750778 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:24.751288 (+ 510us) server_negotiation.cc:299] Negotiation successful
0925 12:21:24.751355 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260925 12:21:24.752228 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.745376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.0.98.61:45129)
0925 12:21:24.745422 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.745423 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.745431 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:21:24.745482 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.745483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.745492 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.745543 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.745966 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.746378 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.746988 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.747180 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.747259 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.749418 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:24.749419 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:24.749419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:24.749425 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:24.750845 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.751258 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.751258 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.751259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.751313 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.751736 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.751737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.751737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.751857 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:24.751985 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:24.752068 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 12:21:24.752149 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 12:21:24.753137 23189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6: Bootstrap starting.
I20260925 12:21:24.753286 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:21:24.753867 23189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:24.754149 23189 log.cc:824] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:24.754014 23013 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3914136d4eaa46ddbf143969a8566501"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47420
I20260925 12:21:24.754961 23189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6: No bootstrap required, opened a new log
Sep 25 12:21:24 dist-test-slave-clmx krb5kdc[22958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 12:21:24.756675 23189 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3914136d4eaa46ddbf143969a8566501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } peers { permanent_uuid: "8e51d1a3bc6341b7a913e813a397cceb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } } peers { permanent_uuid: "ed21138026b24b1e9a4d536d7b6c8bb6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } }
I20260925 12:21:24.756805 23189 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed21138026b24b1e9a4d536d7b6c8bb6, State: Initialized, Role: FOLLOWER
I20260925 12:21:24.756877 23189 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3914136d4eaa46ddbf143969a8566501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } peers { permanent_uuid: "8e51d1a3bc6341b7a913e813a397cceb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } } peers { permanent_uuid: "ed21138026b24b1e9a4d536d7b6c8bb6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } }
I20260925 12:21:24.757151 23189 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:24.757133 23196 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3914136d4eaa46ddbf143969a8566501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } peers { permanent_uuid: "8e51d1a3bc6341b7a913e813a397cceb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } } peers { permanent_uuid: "ed21138026b24b1e9a4d536d7b6c8bb6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } } }
I20260925 12:21:24.757275 23196 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:21:24.758757 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.751161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46345 (local address 127.0.0.1:53306)
0925 12:21:24.751237 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.751247 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:21:24.751277 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:21:24.751497 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:21:24.751498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:21:24.751664 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:21:24.751701 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:24.751721 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.752573 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.752574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:24.753812 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:24.753816 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.754310 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.754311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:24.754369 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.754738 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:21:24.754743 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:21:24.755890 (+ 1147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:21:24.757656 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:24.757657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:24.757658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:24.757860 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:24.758077 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:24.758078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:24.758078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:24.758136 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:24.758456 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:21:24.758456 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:21:24.758495 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 12:21:24.758563 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 12:21:24.758655 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 12:21:24.758780 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.751257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53306 (local address 127.0.98.60:46345)
0925 12:21:24.751384 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.751386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.751392 (+ 6us) server_negotiation.cc:407] Connection header received
0925 12:21:24.751407 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.751407 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.751415 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.751787 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.751824 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.752490 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.753922 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.754212 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.754304 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.755969 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:24.755970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:24.755970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:24.755977 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:24.757539 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.757959 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.757960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.757960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.758009 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.758204 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.758205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.758205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.758338 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:24.758396 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:24.758610 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 12:21:24.758680 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
W20260925 12:21:24.759452 23207 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:21:24.759508 23207 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:21:24.760100 23039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501: Bootstrap starting.
I20260925 12:21:24.760722 23039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:24.760998 23039 log.cc:824] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:24.759078 23121 master_runner.cc:186] Error getting master registration for 127.0.98.62:37011: OK, instance_id { permanent_uuid: "3914136d4eaa46ddbf143969a8566501" instance_seqno: 1790338884411488 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 12:21:24.761176 23121 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 12:21:24.761727 23039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501: No bootstrap required, opened a new log
I20260925 12:21:24.762975 23039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3914136d4eaa46ddbf143969a8566501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } peers { permanent_uuid: "8e51d1a3bc6341b7a913e813a397cceb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } } peers { permanent_uuid: "ed21138026b24b1e9a4d536d7b6c8bb6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } }
I20260925 12:21:24.763082 23039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3914136d4eaa46ddbf143969a8566501, State: Initialized, Role: FOLLOWER
I20260925 12:21:24.763154 23039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3914136d4eaa46ddbf143969a8566501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } peers { permanent_uuid: "8e51d1a3bc6341b7a913e813a397cceb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } } peers { permanent_uuid: "ed21138026b24b1e9a4d536d7b6c8bb6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } }
I20260925 12:21:24.763461 23013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 12:21:24.763404 23210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3914136d4eaa46ddbf143969a8566501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } peers { permanent_uuid: "8e51d1a3bc6341b7a913e813a397cceb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } } peers { permanent_uuid: "ed21138026b24b1e9a4d536d7b6c8bb6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } } }
I20260925 12:21:24.763406 23039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:24.763517 23013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:24.764344 23013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3914136d4eaa46ddbf143969a8566501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } peers { permanent_uuid: "8e51d1a3bc6341b7a913e813a397cceb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } } peers { permanent_uuid: "ed21138026b24b1e9a4d536d7b6c8bb6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } }
I20260925 12:21:24.764561 23013 leader_election.cc:288] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [CANDIDATE]: Term 1 election: Requested vote from peers 8e51d1a3bc6341b7a913e813a397cceb (127.0.98.61:45129), ed21138026b24b1e9a4d536d7b6c8bb6 (127.0.98.60:46345)
I20260925 12:21:24.763510 23210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:21:24.765139 23163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3914136d4eaa46ddbf143969a8566501" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ed21138026b24b1e9a4d536d7b6c8bb6"
I20260925 12:21:24.765252 23163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:21:24.766024 23163 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3914136d4eaa46ddbf143969a8566501 in term 1.
I20260925 12:21:24.766448 22988 leader_election.cc:302] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3914136d4eaa46ddbf143969a8566501, ed21138026b24b1e9a4d536d7b6c8bb6; no voters:
I20260925 12:21:24.766556 23210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:21:24.766714 23210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [term 1 LEADER]: Becoming Leader. State: Replica: 3914136d4eaa46ddbf143969a8566501, State: Running, Role: LEADER
I20260925 12:21:24.766835 23210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3914136d4eaa46ddbf143969a8566501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } peers { permanent_uuid: "8e51d1a3bc6341b7a913e813a397cceb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } } peers { permanent_uuid: "ed21138026b24b1e9a4d536d7b6c8bb6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } }
I20260925 12:21:24.767656 22971 master_runner.cc:186] Error getting master registration for 127.0.98.61:45129: OK, instance_id { permanent_uuid: "8e51d1a3bc6341b7a913e813a397cceb" instance_seqno: 1790338884569930 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 12:21:24.769840 22971 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 12:21:24.765213 23221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:21:24.769901 23221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:21:24.770113 23227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3914136d4eaa46ddbf143969a8566501. Latest consensus state: current_term: 1 leader_uuid: "3914136d4eaa46ddbf143969a8566501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3914136d4eaa46ddbf143969a8566501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } peers { permanent_uuid: "8e51d1a3bc6341b7a913e813a397cceb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } } peers { permanent_uuid: "ed21138026b24b1e9a4d536d7b6c8bb6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } } }
I20260925 12:21:24.770212 23227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:24.770445 23229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:21:24.770709 23229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:21:24.772854 23163 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6 [term 1 FOLLOWER]: Refusing update from remote peer 3914136d4eaa46ddbf143969a8566501: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:21:24.773233 23210 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed21138026b24b1e9a4d536d7b6c8bb6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:21:24.774472 23196 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3914136d4eaa46ddbf143969a8566501. Latest consensus state: current_term: 1 leader_uuid: "3914136d4eaa46ddbf143969a8566501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3914136d4eaa46ddbf143969a8566501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } peers { permanent_uuid: "8e51d1a3bc6341b7a913e813a397cceb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } } peers { permanent_uuid: "ed21138026b24b1e9a4d536d7b6c8bb6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } } }
I20260925 12:21:24.774545 23196 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:21:24.776679 23210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3914136d4eaa46ddbf143969a8566501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3914136d4eaa46ddbf143969a8566501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } peers { permanent_uuid: "8e51d1a3bc6341b7a913e813a397cceb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } } peers { permanent_uuid: "ed21138026b24b1e9a4d536d7b6c8bb6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } } }
I20260925 12:21:24.776759 23210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [sys.catalog]: This master's current role is: LEADER
I20260925 12:21:24.777189 23196 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3914136d4eaa46ddbf143969a8566501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3914136d4eaa46ddbf143969a8566501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } peers { permanent_uuid: "8e51d1a3bc6341b7a913e813a397cceb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } } peers { permanent_uuid: "ed21138026b24b1e9a4d536d7b6c8bb6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } } }
I20260925 12:21:24.777256 23196 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:21:24.777581 23230 mvcc.cc:205] Tried to move back new op lower bound from 7333228072026673152 to 7333228072005685248. Current Snapshot: MvccSnapshot[applied={T|T < 7333228072026673152}]
I20260925 12:21:24.778048 23234 mvcc.cc:205] Tried to move back new op lower bound from 7333228072026673152 to 7333228072005685248. Current Snapshot: MvccSnapshot[applied={T|T < 7333228072026673152}]
I20260925 12:21:24.780449 23229 catalog_manager.cc:1347] Generated new cluster ID: d0ac73f51b554f678fe8f6bfd8acfe5d
I20260925 12:21:24.780489 23229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:21:24.786617 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.772218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.0.98.61:45129)
0925 12:21:24.772332 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.772334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.772593 (+ 259us) server_negotiation.cc:407] Connection header received
0925 12:21:24.772829 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.772830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.772839 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.772867 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.776048 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.776703 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.777651 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.777924 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.777960 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.781227 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:24.781228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:24.781229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:24.781235 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:24.782749 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.785176 (+ 2427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.785177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.785177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.785229 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.785685 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.785686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.785686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.785808 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:24.785884 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:24.786457 (+ 573us) server_negotiation.cc:299] Negotiation successful
0925 12:21:24.786531 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260925 12:21:24.787361 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.772372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53314 (local address 127.0.98.60:46345)
0925 12:21:24.772469 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.772471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.772676 (+ 205us) server_negotiation.cc:407] Connection header received
0925 12:21:24.772920 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.772921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.772931 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.772960 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.774338 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.775043 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.778396 (+ 3353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.778673 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.778710 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.783144 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:24.783145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:24.783145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:24.783152 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:24.784764 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.785423 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.785424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.785425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.785480 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.786044 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.786045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.786045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.786175 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:24.786256 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:24.787215 (+ 959us) server_negotiation.cc:299] Negotiation successful
0925 12:21:24.787286 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 12:21:24.799309 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.771825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47426 (local address 127.0.98.62:37011)
0925 12:21:24.772262 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.772264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.778381 (+ 6117us) server_negotiation.cc:407] Connection header received
0925 12:21:24.778519 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.778520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.778540 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.778566 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.791791 (+ 13225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.792473 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.793473 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.793769 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.793847 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.796073 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:24.796085 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:24.796085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:24.796091 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:24.797755 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.798329 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.798330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.798330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.798393 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.798673 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.798674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.798674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.798780 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:24.798846 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:24.799160 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 12:21:24.799229 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418}
I20260925 12:21:24.802227 23114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } attempt: 3
Sep 25 12:21:24 dist-test-slave-clmx krb5kdc[22958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 12:21:24.806945 23229 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:21:24.807751 23229 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:21:24.810920 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.802428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46345 (local address 127.0.0.1:53330)
0925 12:21:24.802496 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 12:21:24.802505 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:21:24.802538 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:21:24.802757 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:21:24.802758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:21:24.802903 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:21:24.802938 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:24.802940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.803905 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.803906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:24.804707 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:21:24.804709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.805274 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.805275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:21:24.805343 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.805795 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:21:24.805800 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:21:24.807888 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:21:24.809797 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:24.809798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:24.809798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:24.810024 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:24.810282 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:21:24.810282 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:21:24.810283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:21:24.810348 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:21:24.810637 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:21:24.810638 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:21:24.810691 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 12:21:24.810748 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 12:21:24.810832 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 12:21:24.811275 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:24.802600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.0.98.60:46345)
0925 12:21:24.802632 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.802634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.802640 (+ 6us) server_negotiation.cc:407] Connection header received
0925 12:21:24.802655 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.802656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.802663 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.802704 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.803013 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.803792 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.804847 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:24.805159 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:24.805213 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:24.807980 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:21:24.807981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:21:24.807982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:21:24.807988 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:21:24.809679 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.810136 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.810138 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.810138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.810201 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:21:24.810422 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:21:24.810423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:21:24.810423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:21:24.810544 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:21:24.810596 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:24.811129 (+ 533us) server_negotiation.cc:299] Negotiation successful
0925 12:21:24.811196 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:21:24.812530 23114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb: Bootstrap starting.
I20260925 12:21:24.813238 23114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb: Neither blocks nor log segments found. Creating new log.
I20260925 12:21:24.813513 23114 log.cc:824] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb: Log is configured to *not* fsync() on all Append() calls
I20260925 12:21:24.815120 23114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb: No bootstrap required, opened a new log
I20260925 12:21:24.816991 23114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3914136d4eaa46ddbf143969a8566501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } peers { permanent_uuid: "8e51d1a3bc6341b7a913e813a397cceb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } } peers { permanent_uuid: "ed21138026b24b1e9a4d536d7b6c8bb6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } }
I20260925 12:21:24.817117 23114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e51d1a3bc6341b7a913e813a397cceb, State: Initialized, Role: FOLLOWER
I20260925 12:21:24.817181 23114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3914136d4eaa46ddbf143969a8566501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } peers { permanent_uuid: "8e51d1a3bc6341b7a913e813a397cceb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } } peers { permanent_uuid: "ed21138026b24b1e9a4d536d7b6c8bb6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } }
I20260925 12:21:24.817217 23229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501: Generated new TSK 0
I20260925 12:21:24.817420 23229 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:21:24.817454 23114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:21:24.818638 23088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb [term 0 FOLLOWER]: Advancing to term 1
W20260925 12:21:24.819057 23251 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:21:24.819152 23251 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:21:24.819384 23088 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb [term 1 FOLLOWER]: Refusing update from remote peer 3914136d4eaa46ddbf143969a8566501: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:21:24.819710 23237 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e51d1a3bc6341b7a913e813a397cceb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:21:24.821038 23256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3914136d4eaa46ddbf143969a8566501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3914136d4eaa46ddbf143969a8566501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } peers { permanent_uuid: "8e51d1a3bc6341b7a913e813a397cceb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } } peers { permanent_uuid: "ed21138026b24b1e9a4d536d7b6c8bb6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } } }
I20260925 12:21:24.821179 23256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:21:24.821261 23256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3914136d4eaa46ddbf143969a8566501. Latest consensus state: current_term: 1 leader_uuid: "3914136d4eaa46ddbf143969a8566501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3914136d4eaa46ddbf143969a8566501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } peers { permanent_uuid: "8e51d1a3bc6341b7a913e813a397cceb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } } peers { permanent_uuid: "ed21138026b24b1e9a4d536d7b6c8bb6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } } }
I20260925 12:21:24.821347 23256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:21:24.821365 23086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3914136d4eaa46ddbf143969a8566501" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8e51d1a3bc6341b7a913e813a397cceb"
I20260925 12:21:24.822517 23086 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3914136d4eaa46ddbf143969a8566501 for term 1 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260925 12:21:24.822649 23254 mvcc.cc:205] Tried to move back new op lower bound from 7333228072206696448 to 7333228072005685248. Current Snapshot: MvccSnapshot[applied={T|T < 7333228072026673152}]
I20260925 12:21:24.821410 23256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3914136d4eaa46ddbf143969a8566501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3914136d4eaa46ddbf143969a8566501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } peers { permanent_uuid: "8e51d1a3bc6341b7a913e813a397cceb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } } peers { permanent_uuid: "ed21138026b24b1e9a4d536d7b6c8bb6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } } }
I20260925 12:21:24.827767 23256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb [sys.catalog]: This master's current role is: FOLLOWER
W20260925 12:21:24.838392 23042 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:24.837020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47428 (local address 127.0.98.62:37011)
0925 12:21:24.837054 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.837055 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.837825 (+ 770us) server_negotiation.cc:407] Connection header received
0925 12:21:24.837910 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.837911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.837921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.837946 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.838323 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47428: BlockingRecv error: recv got EOF from 127.0.0.1:47428 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 12:21:24.839038 23118 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:24.837368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44996 (local address 127.0.98.61:45129)
0925 12:21:24.837411 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.837413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.838629 (+ 1216us) server_negotiation.cc:407] Connection header received
0925 12:21:24.838702 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.838703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.838711 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.838737 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.838988 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44996: BlockingRecv error: recv got EOF from 127.0.0.1:44996 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 12:21:24.839646 23194 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:24.837614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53334 (local address 127.0.98.60:46345)
0925 12:21:24.837644 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.837647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.839198 (+ 1551us) server_negotiation.cc:407] Connection header received
0925 12:21:24.839300 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.839301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.839309 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.839334 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.839574 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53334: BlockingRecv error: recv got EOF from 127.0.0.1:53334 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 12:21:24.841570 23118 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:24.840897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.0.98.61:45129)
0925 12:21:24.840925 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.840927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.841111 (+ 184us) server_negotiation.cc:407] Connection header received
0925 12:21:24.841201 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.841202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.841216 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.841240 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.841531 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45012: BlockingRecv error: recv got EOF from 127.0.0.1:45012 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260925 12:21:24.844400 23118 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:24.843685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.0.98.61:45129)
0925 12:21:24.843724 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.843726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.843931 (+ 205us) server_negotiation.cc:407] Connection header received
0925 12:21:24.844011 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.844011 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.844021 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.844045 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.844362 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45028: BlockingRecv error: recv got EOF from 127.0.0.1:45028 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 12:21:24.847955 23118 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:24.847317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.0.98.61:45129)
0925 12:21:24.847345 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.847346 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.847521 (+ 175us) server_negotiation.cc:407] Connection header received
0925 12:21:24.847601 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.847601 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.847609 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.847672 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.847921 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45042: BlockingRecv error: recv got EOF from 127.0.0.1:45042 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260925 12:21:24.853657 23118 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:24.852964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.0.98.61:45129)
0925 12:21:24.852993 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.852995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.853215 (+ 220us) server_negotiation.cc:407] Connection header received
0925 12:21:24.853305 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.853306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.853314 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.853342 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.853619 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45054: BlockingRecv error: recv got EOF from 127.0.0.1:45054 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 12:21:24.863401 23118 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:24.862642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45070 (local address 127.0.98.61:45129)
0925 12:21:24.862673 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.862675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.862906 (+ 231us) server_negotiation.cc:407] Connection header received
0925 12:21:24.863001 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.863002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.863011 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.863039 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.863352 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45070: BlockingRecv error: recv got EOF from 127.0.0.1:45070 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 12:21:24.881372 23118 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:24.880565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45080 (local address 127.0.98.61:45129)
0925 12:21:24.880597 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.880599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.880864 (+ 265us) server_negotiation.cc:407] Connection header received
0925 12:21:24.880961 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.880962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.880971 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.880999 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.881327 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45080: BlockingRecv error: recv got EOF from 127.0.0.1:45080 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 12:21:24.915784 23118 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:24.914833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45086 (local address 127.0.98.61:45129)
0925 12:21:24.914876 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.914878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.915153 (+ 275us) server_negotiation.cc:407] Connection header received
0925 12:21:24.915253 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.915255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.915267 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.915299 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.915723 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45086: BlockingRecv error: recv got EOF from 127.0.0.1:45086 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 12:21:24.981719 23118 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:24.981064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.0.98.61:45129)
0925 12:21:24.981123 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:24.981126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:24.981260 (+ 134us) server_negotiation.cc:407] Connection header received
0925 12:21:24.981319 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:24.981320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:24.981331 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:24.981384 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:24.981662 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45088: BlockingRecv error: recv got EOF from 127.0.0.1:45088 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 12:21:25.111524 23118 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:25.110911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.0.98.61:45129)
0925 12:21:25.110977 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:25.110979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:25.111074 (+ 95us) server_negotiation.cc:407] Connection header received
0925 12:21:25.111140 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:25.111140 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:25.111162 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:25.111217 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:25.111479 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45094: BlockingRecv error: recv got EOF from 127.0.0.1:45094 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 12:21:25.207832 23210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3914136d4eaa46ddbf143969a8566501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3914136d4eaa46ddbf143969a8566501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } peers { permanent_uuid: "8e51d1a3bc6341b7a913e813a397cceb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45129 } } peers { permanent_uuid: "ed21138026b24b1e9a4d536d7b6c8bb6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46345 } } }
I20260925 12:21:25.207976 23210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3914136d4eaa46ddbf143969a8566501 [sys.catalog]: This master's current role is: LEADER
W20260925 12:21:25.369318 23118 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:21:25.368714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45098 (local address 127.0.98.61:45129)
0925 12:21:25.368780 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:25.368782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:25.368824 (+ 42us) server_negotiation.cc:407] Connection header received
0925 12:21:25.368890 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:25.368891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:25.368901 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:25.368943 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:21:25.369242 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45098: BlockingRecv error: recv got EOF from 127.0.0.1:45098 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 12:21:25.414433 23034 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.98.62:37011
I20260925 12:21:25.760011 23207 catalog_manager.cc:1259] Loaded cluster ID: d0ac73f51b554f678fe8f6bfd8acfe5d
I20260925 12:21:25.760084 23207 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6: loading cluster ID for follower catalog manager: success
I20260925 12:21:25.760824 23207 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6: acquiring CA information for follower catalog manager: success
I20260925 12:21:25.761106 23207 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ed21138026b24b1e9a4d536d7b6c8bb6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 12:21:25.819566 23251 catalog_manager.cc:1259] Loaded cluster ID: d0ac73f51b554f678fe8f6bfd8acfe5d
I20260925 12:21:25.819664 23251 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb: loading cluster ID for follower catalog manager: success
I20260925 12:21:25.820447 23251 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb: acquiring CA information for follower catalog manager: success
I20260925 12:21:25.820744 23251 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8e51d1a3bc6341b7a913e813a397cceb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 12:21:25.884945 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:25.882410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45102 (local address 127.0.98.61:45129)
0925 12:21:25.882515 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:25.882517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:25.882576 (+ 59us) server_negotiation.cc:407] Connection header received
0925 12:21:25.882655 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:25.882656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:25.882695 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:25.882825 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:21:25.882999 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:25.883475 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:25.884252 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:25.884437 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:25.884489 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:25.884629 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:21:25.884729 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:21:25.884787 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:25.884821 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 12:21:25.884838 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":63,"threads_started":1}
I20260925 12:21:25.888650 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:25.886577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.0.98.60:46345)
0925 12:21:25.886679 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:25.886681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:25.886715 (+ 34us) server_negotiation.cc:407] Connection header received
0925 12:21:25.886747 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:25.886748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:25.886779 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:25.886834 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:21:25.886914 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:25.887367 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:25.888065 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:25.888244 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:25.888303 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:25.888412 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:21:25.888494 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:21:25.888545 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:25.888576 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 12:21:25.888596 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":63,"threads_started":1}
I20260925 12:21:25.892903 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:25.890192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45116 (local address 127.0.98.61:45129)
0925 12:21:25.890223 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:25.890225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:25.890232 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:21:25.890254 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:25.890255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:25.890274 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:25.890326 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:21:25.890507 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:25.891227 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:25.892189 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:25.892460 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:25.892525 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:25.892645 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:21:25.892670 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:21:25.892712 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:25.892819 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 12:21:25.892839 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 12:21:25.892951 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:25.890304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53346 (local address 127.0.98.60:46345)
0925 12:21:25.890349 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:25.890351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:25.890493 (+ 142us) server_negotiation.cc:407] Connection header received
0925 12:21:25.890570 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:25.890571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:25.890580 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:25.890634 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:21:25.890719 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:25.891377 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:25.892323 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:25.892486 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:25.892553 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:25.892721 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:21:25.892746 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:21:25.892803 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:25.892858 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 12:21:25.892875 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:21:25.893854 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:21:25.890042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47438 (local address 127.0.98.62:37011)
0925 12:21:25.890407 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0925 12:21:25.890409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:21:25.890417 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:21:25.890461 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:21:25.890462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:21:25.890508 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:21:25.890579 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:21:25.891447 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:25.891970 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:25.892987 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:21:25.893290 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:21:25.893357 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:21:25.893530 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:21:25.893624 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:21:25.893673 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 12:21:25.893721 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 12:21:25.893740 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":66,"threads_started":1}
I20260925 12:21:25.895340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 22971
I20260925 12:21:25.900523 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 23046
I20260925 12:21:25.905093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKTlv5c/build/release/bin/kudu with pid 23121
2026-09-25T12:21:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2699 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5394 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (260819 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST